AUTOMORPHIC INTEGRAL TRANSFORMS FOR 
CLASSICAL GROUPS I: ENDOSCOPY 
CORRESPONDENCES 

DIHUA JIANG 

Abstract. A general framework of constructions of endoscopy 
correspondences via automorphic integral transforms for classical 
groups is formulated in terms of the Arthur classification of the 
discrete spectrum of square-integrable automorphic forms. This 
suggests a principle, which is called the (r, 6)-theory of automor- 
phic forms of classical groups, to reorganize and extend the series 
of work of Piatetski-Shapiro, Rallis, Kudla and others on standard 
^-functions of classical groups and theta correspondence. 



1. Introduction 

Automorphic forms are transcendental functions with abundant sym- 
metries and fundamental objects to arithmetic and geometry. In the 
theory of automorphic forms, it is an important and difficult problem 
to construct explicitly automorphic forms with specified properties, in 
particular, to construct cuspidal automorphic forms of various types. 

Ilya Piatetski-Shapiro is a pioneer to use representation theory to 
study automorphic forms. The main idea is to construct certain mod- 
els of representation-theoretic nature to obtain cuspidal automorphic 
forms, that is, to explicitly realize cuspidal automorphic forms in the 
space of functions over certain geometric spaces. The celebrated theo- 
rem of Gelfand and Piatetski-Shapiro shows that all cuspidal automor- 
phic forms are rapidly decreasing on a fundamental domain D when ap- 
proaching the cusps, and hence can be realized discretely in the space 
L 2 (JD>) of all square integrable functions on D. Therefore, it is funda- 
mental to understand the space L 2 (©) or more precisely the discrete 
spectrum of L 2 (D). 
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In the modern theory of automorphic forms, this problem is formu- 
lated as follows. Let G be a reductive algebraic group defined over a 
number field (or a global field) F and consider the quotient space 

X G = Z G (A)G(F)\G(A) 

where A is the ring of adeles of F and Z G is the center of G. As a con- 
sequence of the reduction theory in the arithmetic theory of algebraic 
groups, this space X G has finite volume with respect to the canonical 
quotient measure. Geometrically X G is a modern replacement of the 
classical locally symmetric spaces which are the land for classical auto- 
morphic forms. It is fundamental to understand the space L 2 (X G ) of 
all square integrable functions. It is clear that under the right trans- 
lation by G(A), the space L 2 (X G ) is unitary representation of G(A) 
with respect to the L 2 -inner product. The theorem of Gelfand and 
Piatetski-Shapiro says that all irreducible cuspidal automorphic repre- 
sentations of G(A) (with trivial central character) can be realized as 
irreducible submodules of L 2 (X G ). Combining with the Langlands the- 
ory of Eisenstein series, one can state the following fundamental result 
in the theory of automorphic forms. 

Theorem 1.1 (Gelfand, Piatetski-Shapiro; Langlands). The discrete 
spectrum of the space of square integrable automorphic functions has 
the following decomposition 

l Lc( x g) = © 7re g(A)m disc (7r)y 7r 

with finite multiplicity md isc (7r) for all irreducible unitary representa- 
tions (tt, Vn) of G(A), where G(A) is the unitary dual of G(A). 

One of the remaining fundamental problems concerning L/^ sc (Xq s ) is 
to understand the multiplicity mdi S c(7r) i n general. 

When G = GL„, the celebrated theorems of Shalika ( |S174j ) and 
of Piatetski-Shapiro ( |PS79j ) for cuspidal spectrum; and of Moeglin 
and Waldspurger QMW89J) for residual spectrum assert that for any 
irreducible unitary representation 7r of GL n (A) has multiplicity m&acfa) 
at most one. However, when G 7^ GL n , the situation could be very 
different. 

For instance, when G = SL n with n > 3, D. Blasius finds a family 
of cuspidal automorphic representations with higher multiplicity, i.e. 
^idisc(Tr) > 1 QB194]). See also the work of E. Lapid ( |Lp99| ). However, 
when n = 2 the multiplicity is one, which is proved by D. Ramakrishnan 
( [RmOO] ) based on an earlier work of Labesse and Langlands ( |LL79j ) . 
More interestingly, when G = G2, the F-split exceptional group of type 
G2, Gan, Gurevich and the author show that there exist a family of 
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cuspidal automorphic representations of (^(A), whose cuspidal multi- 
plicity can be as high as possible ( |GGJ02j ). and see |Gn05] for more 
complete result. From the argument of |GGJ02j . such a result can also 
be expected for other exceptional groups. On the other hand, it does 
not happen for F-quasisplit classical groups following Arthur's theo- 
rem in his forthcoming book ( |Arl2j ). but when classical groups are 
not F-quasisplit, the multiplicity could still be higher according to the 
work of J.-S. Li ( [Li95J and |Li97j ). Now, following the Arthur mul- 
tiplicity formula ( |Arl2j and |Mkl2j ). the multiplicity in the discrete 
spectrum of F-quasisplit classical groups is at most one in general, 
except for certain cases of even special orthogonal groups where the 
discrete multiplicity could be two. 

When G = GL ra , the discrete spectrum was completely determined 
by Moeglin and Waldspurger in 1989 QMW89J), which confirms a con- 
jecture of Jacquet in 1983. For G to be a F-quasisplit classical groups, 
the stable trace formula of Arthur is able to give a complete descrip- 
tion of the discrete spectrum, up to global Arthur packets, in terms 
of a certain family of automorphic representations of GL, based on 
the fundamental work of B.-C. Ngo ( |N10j ) and of G. Laumon, P.-H. 
Chaudouard, J.-L. Waldspurger, and others (see the references within 
|Arl2] and |Mkl2j ). If G is not F-quasisplit (i.e. general inner forms), 
much more work needs to be done ( |Arl2t Chapter 9]). It is far from 
reach if G is of exceptional type. 

One of the great outcomes from the trace formula approach to un- 
derstand the structure of the discrete spectrum of classical groups is 
the discovery of the endoscopy structure of automorphic forms by R. 
Langlands ( |L79j and |L83j ). the twisted version of which is referred to 
the book of R. Kottwitz and D. Shelstad ( |KtS 99j). The trace formula 
method confirms the existence of the endoscopy structure of automor- 
phic forms on classical groups, which leads a uniform proof of the exis- 
tence of the weak functorial transfer from F-quasisplit classical groups 
to the general linear groups for automorphic representations occurring 
in the discrete spectrum of classical groups, among other fundamentally 
important results ( |Arl2j . and also |Mkl2]), based on the fundamental 
work of Laumon, Ngo, Waldspurger, Chaudouard, and others. 

Endoscopy transfers are the Langlands functorial transfers, which 
indicates that automorphic representations G(A) may come from its 
endoscopy groups. This is a special case of the general Langlands func- 
toriality conjecture ( |L70j ). In addition to the trace formula approach, 
there have been other methods to prove the existence or to construct 
explicitly various cases of the Langlands functorial transfers. 



4 



DIHUA JIANG 



One of the major methods is to establish the Langlands functorial 
transfers from classical groups to general linear groups via the Con- 
verse Theorem of Cogdell and Piatetski-Shapiro ( [CPS94] ) combined 
with the analytic properties of tensor product L-functions of classical 
groups and general linear groups. This method has been successful for 
generic cuspidal automorphic representations of F-quasisplit classical 
groups ( |CKPSS04j . [UPSSllj . |KK05j . and also see |A"S06] for spinor 
group case) and has been extended to prove the existence of the n- 
symmetric power liftings of GL(2) for n = 2,3,4 ( [GTJ78] . |KmS02] 
and |Km03j ) . As outlined by D. Soudry in |Sd06| . this method can 
be essentially used to establish the Langlands functorial transfers (the 
strong version, i.e. compatible with the local Langlands transfer at all 
local places) from classical groups to general linear groups for all cusp- 
idal automorphic representations, as long as the analytic properties of 
tensor product L- functions of classical groups and general linear groups 
can be established in this generality. Some recent progress in this as- 
pect is referred to the work of Pitale, Saha, and Schmidt ( jPSSllj ) and 
of Lei Zhang and the author ( |JZ12j and also [Snl2j and [Snl3j ) . 

Another method to establish certain cases of the existence of the 
Langlands functoriality is to use the relative trace formula approach 
which was introduced by H. Jacquet. See |Jc97] for discussions in this 
aspect and jJcOi] and [MRlOj for more recent progress. 

For the Langlands functorial transfers beyond the endoscopy theory, 
it is referred to the work of Langlands ( |L04j ) and the more recent work 
of Frenkel, Langlands and Ngo ( |FLN10j and [FNllJ). An alternative 
approach is taken by L. Lafforgue ( |LflO] and |Lfl2j . and also |Jcl2j ). 

The remaining methods are more constructive in nature. 

First, the theta correspondence for reductive dual pairs in the 
sense of Howe is a method to construct automorphic forms using the 
theta functions built from the Weil representation on the Schrodinger 
model. The method of using classical theta functions to construct 
automorphic forms goes back at least to the work of C. Siegel ( |Sg66| ) 
and the work of A. Weil ( I W'KiTj and |W165j ). The fundamental work 
of Howe (|Hw79j) started the representation-theoretic approach in this 
method guided by the idea from Invariant Theory. 

This idea (and the method) has been extended a similar construc- 
tions for exceptional groups and reductive dual pairs in the 1990 's, and 
is usually called the exceptional theta correspondence. The main 
work includes that of Kazhdan and Savin QKzS90j), that of Ginzburg, 
Rallis, and Soudry ([GRS97]), and many others. 

More recently, a striking method to construct more general automor- 
phic forms was carried by Ginzburg, Rallis, and Soudry in 1999 (called 
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the automorphic descent method), which extends the classical ex- 
amples of Eichler and Zagier QEZ85J) and of I. Piatetski-Shapiro on 
Sp(4) ( |PS83j ) to general F-quasisplit classical groups and establishes 
the relation between their constructions and the Langlands functorial- 
ity between classical groups and general linear groups. This construc- 
tion is discussed in general in their recent book ( |GRSllj ) and will be 
reviewed with some details in Section 6, including some extension of 
the methods through the work of Ginzburg, Soudry, Baiying Liu, Lei 
Zhang and the author QGJS12] . |GJS13] . [JL13] . [JZT3] and |Liul3] K 

The objective here is to reformulate and extend the previous known 
cases of constructions and the related theory in a much more general 
framework in terms of the Arthur classification of the discrete spectrum 
of classical groups, and suggest a new theory, called (r, 6)-theory, of 
automorphic forms for further exploration and investigation. 

One of the key ingredients of the (r, 6)-theory is to formulate ex- 
plicit constructions of the elliptic endoscopy correspondences (includ- 
ing transfers and descents), via automorphic integral transforms 
with the kernel functions. Such general constructions extends almost 
all previous known constructions in the theory of automorphic forms. 
Some particular cases were formulated by Ginzburg for F-split classical 
groups in |G08| and |G12] for constructions of special CAP automor- 
phic representations in the sense of Piatetski-Shapiro, without referring 
to the Arthur classification theory ( |Arl2j ). However, it is much better 
understood if they are formulated in terms of Arthur's classification of 
the discrete spectrum of classical groups ([Arl2] and |Mkl2j ). This is 
the main point of the current paper, which extends the case of certain 
orthogonal groups considered in the Oberwolfach report of the author 

(GEU). 

In this introduction, G is assumed to be an F-quasisplit classical 
group, which is not a unitary group. The details about unitary groups 
will be covered in the rest of the paper. Let ip £ ^2 (GO be a global 
Arthur parameter for G, following the notation in [Arl2j . Assume that 
ip is not simple. This means that ip can be expressed as a formal sum 

(1.1) ^ = ^lffl^2, 

and there are two classical groups G = G(ipi) and H = G(^ 2 ) de- 
termined by ipi an d i>2, respectively, such that ipi £ ^(Go) and 
ip2 £ ^2{H) and Go x H is an elliptic endoscopy group of G. The key 
point here is to use ipi (and the structures of H and G) to construct 
a family of automorphic kernel functions )C$ Vt H,G(h>, g), which are 
automorphic functions on H(A) x G(A). 
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Conjecture 1.2 (Main Conjecture on Endoscopy Correspondences). 
Assume that if) = if>i EH if>2 is a global Arthur parameter for G such that 
G = G{jpi) and H = G{ip2) are twisted endoscopy groups associated 
to ipi and if> 2 , respectively, so that Go x H is an elliptic endoscopy 
group of G associated to if). Let o and tt be irreducible automorphic 
representations of H(A) and G(A) occurring in the discrete spectrum 
respectively. Then there exists a family of automorphic kernel functions 
K.^ i; h,g{K 9) built from ipi, and the structures of H and G, such that 
for (f a Ec and tp^ G tt, if the following integral 



is convergent and nonzero, then a G n^ 2 (e^, 2 ) if and only if tt £ 
IL$(e$) where H^ 2 (e^ 2 ) and Il^(e^) are the corresponding automorphic 
L 2 -packets, the definition of which is given in Section 2.1. 

It is clear that the integral in (11.21) converges absolutely if a and 
tt are cuspidal; otherwise, a suitable regularization (by using Arthur 
truncation, for instance) is needed. This integral also produces two- 
way constructions or integral transforms as follows. 

Let a be an irreducible cuspidal automorphic representation of H(A). 
Define 



Then this integral is absolutely convergent and defines an automor- 
phic function on G(A). Denote by T^^ic) the space generated by all 
automorphic functions T^^h^o)^) and call T^ 3i ,h(c) the endoscopy 
transfer from H to G via if)\ or the endoscopy transfer from Go x H 
to G. When a is not cuspidal, the integral in (11.31) may not be conver- 
gent and certain regularization is needed. 



Conversely, let tt be an irreducible cuspidal automorphic representa- 
tion of G(A). Define 



Then this integral is absolutely convergent and defines an automorphic 
function on H(A). Denote by X>^, 1)G (7r) the space generated by all 
automorphic functions V^ lt G(fn)(h) and call T>^ u g(tt) the endoscopy 
descent from G to H via ip±. When tt is not cuspidal, the integral in 
(11.41) may not be convergent and certain regularization is needed. 

The goal is to investigate the automorphic representations ^^^(cr) 
of G(A) and V^ ltG (iT) of H(A), respectively. From Conjecture ll.2[ 



(1.2) 




(1.3) 




(1.4) 
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one may expect to produce automorphic representations in the global 
Arthur packet n^, lffl ^ 2 of G(A) from the endoscopy transfer T^ lt H(ff) if 
a runs in the automorphic L 2 -packet n^ 2 (e^ 2 ); and to produce auto- 
morphic representations in the global Arthur packet U^ 2 of if (A) from 
the endoscopy descent V^ lt c(^) if ^ runs in the automorphic L 2 -packet 

According to the structure of global Arthur parameters, any ip e 
\E' 2 (G) can be expressed as a formal sum 

(1.5) V = (7i,6 1 )ffl(r 2 ,6 2 )ffl...ffl(r r ,6 r ) 

of simple global Arthur parameters (r 1; bi), (r 2 , 62), ■ ■ • , and (x r ,6 r ), 
where t% are irreducible unitary self-dual cuspidal automorphic rep- 
resentations of GL ai (A) and b{ > 1 are integers representing the b{- 
dimensional irreducible representation V{ of SL2(C) for all i. In [Arl2j . 
(tj, bi) are replaced by Ti S ^, respectively. It is natural to start the 
investigation with ipi = (r, b) in Conjecture 11.21 i.e. 

i> = {r,b)S^. 

The (r, 6)-theory of automorphic forms is to understand Conjecture 
11.21 with ipi = (r, b) and related problems, based on or guided by 
the Arthur classification of the discrete spectrum for classical groups. 
The core problem is to understand the spectral structures of T^ u h{°') 
and V^ 1!G (tt); and to characterize the non- vanishing of T^ lt H(cr) and 
^V.Gty) i n terms of certain basic invariants of automorphic forms. It 
is not clear how to use the basic invariants from the trace formula, 
i.e., the stable orbital integrals, to the study of Conjecture ll.2[ because 
technically, it is not clear how to relate the integrals in Conjecture 11.21 
to the corresponding stable orbital integrals, although this is the most 
natural way to think of in the current setting. 

When t is a quadratic character \ of GLi(A), the (x, &)-theory is es- 
sentially about the theta correspondence for reductive dual pairs. The 
basic invariants used by Rallis in his program ([R187] and [R191J) are 
the poles of the twisted standard L-functions through the doubling in- 
tegrals of Piatetski-Shapiro and Rallis ([GPSR87J). For the general 
(r, 6)-theory, the first basic invariant considered is the structure of 
Fourier coefficients of automorphic forms and automorphic represen- 
tations. As the theory develops, other invariants like poles or special 
values of certain family of automorphic L-functions, and various pe- 
riods of automorphic forms will be employed. In this generality, the 
Fourier coefficients of automorphic forms are defined in terms of nilpo- 
tent orbits in the corresponding Lie algebra and are natural invariants 



s 



DIHUA JIANG 



of automorphic representations close to the type of integrals in ( 11. 2(1 . 
Section H] is to discuss the basic theory in this aspect. 

The Arthur classification of the discrete spectrum for F-quasisplit 
classical groups is reviewed briefly in Section |2j where the basic en- 
doscopy structures of each global Arthur packet is discussed. This leads 
to the explicit constructions of the automorphic kernel functions. 
More precisely, the constructions are formulated for the case when tpi 
is a simple global Arthur parameter, say, of form (r, b), where r is 
an irreducible self-dual unitary cuspidal automorphic representation 
of GL a (A) and b > 1 is an integer. Because of the parity structure of 
simple Arthur parameters, this leads to two different types of construc- 
tions: one uses the generalized Bessel- Fourier coefficients in Section 15.31 
and the other uses the generalized Fourier- Jacobi coefficients in Section 
I5.4[ based on the explicit classification of endoscopy structure for all 
F-quasisplit classical groups in Section 3. In Section 6, the discussions 
are given to address how the theory of automorphic descent method 
can be extended from the viewpoint of the (r, 6)-theory to obtain re- 
fined structure about the global Arthur packets of simple type. This 
includes the work of Ginzburg, Soudry and the author ( |GJS12j ). the 
thesis of Baiying Liu ( |Liul3j ) and a more recent work of Liu and the 
author ( |JL13j ). The particular case of (r, 6)-theory with r being a 
quadratic character x of GLi and its relation to the theory of theta 
correspondence will be discussed in Section 7. Section 8 will address 
the general formulation of the (r, 6)-theory and basic problems. The 
final section is to address other related constructions and other related 
topics in the theory of automorphic forms. 

The author is strongly influenced by the idea of the automorphic de- 
scent method of Ginzburg, Rallis and Soudry on the topics discussed in 
the paper, and benefits greatly from his long term collaboration with 
Ginzburg and Soudry on various research projects related to topics dis- 
cussed here. The main idea of the author contributing to the theory is 
to formulate the general framework of the constructions of endoscopy 
correspondences via automorphic integral transforms in terms of the 
Arthur classification of the discrete spectrum of F-quasisplit classical 
groups. This idea came up to the author after reading the Clay lecture 
notes of Arthur ( |Ar05j ) . who sent it to me before it got published. In 
particular, this idea of the author got developed during the two-month 
visit in Paris in 2006, which was arranged by B.-C. Ngo for a one-month 
period in University of Paris-Sud and by M. Harris for a one-month pe- 
riod in University of Paris 7. Since then, the author has been invited 
to talk about the constructions and the related theory at many places, 
including Paris, Jerusalem, Toronto, New York, Vienna, Oberwolfach, 
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Beijing, Philadelphia, Chicago, Boston, New Haven, and so on. The 
author would like to take this opportunity to express his appreciation 
to all the invitations and supports he received over the years. In partic- 
ular, the author would like to thank J. Arthur, M. Harris, C. Moeglin, 

B. -C. Ngo, and D. Vogan for helpful conversations and suggestions, to 
thank D. Ginzburg and D. Soudry for productive collaboration on rel- 
evant research projects over the years, and to thank Baiying Liu and 
Lei Zhang for joining this project recently, which brings progress in 
various aspects of the project. The guidance of I. Piatetski-Shapiro 
and S. Rallis which brought the author to this subject can be clearly 
noticed throughout the whole paper and will continue to influence him 
in his future work. The work of the author has been supported in part 
by the NSF Grant DMS-0400414, DMS-0653742, and DMS-1001672, 
over the years. Finally, the author would like to thank R. Howe for his 
leadership in the organization of the conference: Automorphic Forms 
and Related Geometry: assessing the legacy of I.I. Piatetski-Shapiro, 
at Yale University, April, 2012, which makes it possible to write up this 
paper for the conference proceedings. 

2. Arthur Parametrization of Discrete Spectrum 

Let F be a number field and E be a quadratic extension of F, whose 
Galois group is denoted by T E / F = {l,c}. Take G n to be either the F- 
quasisplit unitary groups U2n = U^/i?(2n) or U2n+i = U^/i?(2n + 1) of 
hermitian type, the F-split special orthogonal group S0 2 „+i, the sym- 
plectic group Sp 2n , or the F-quasisplit even special orthogonal group 
S0 2n - Let F' be a number field, which is F if G n is not a unitary group, 
and is E if G n is a unitary group. Denote by Rjr//_p(n) := R j p//i?(GL n ) 
the Weil restriction of the GL n from F' to F. 

Following Chapter 1 of Arthur's book ( |Arl2j ) and also the work of 

C. -P. Mok ( |Mkl2| ). take the closed subgroup G(A) 1 of G(A) given by 

(2.1) G(A) 1 := {x e G(A) | \ X (x)\ A = 1, V X e X(G) F } 

where X(G)p is the group of all F-rational characters of G. Since 
G(F) is a discrete subgroup of G(A) 1 and the quotient G(F)\G(A) 1 
has finite volume with respect to the canonical Haar measure, consider 
the the space of all square integrable functions on G(F)\G'(A) 1 , which 
is denoted by L 2 (G(F)\G(AY). It has the following embedded, right 
G (A) ^invariant Hilbert subspaces 

(2-2) 4 2 usp (G(F)\G(A) 1 ) c Ll sc (G(F)\G(Ay) c L 2 (C7(F)\C7(A) 1 ) 

where L 2 lisc (G(F)\L7(A) 1 ) is called the discrete spectrum of G(A) 1 , 
which decomposes into a direct sum of irreducible representations of 
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G(A) 1 ; and Ll usp (G(F)\G(A) 1 ) is called the cuspidal spectrum of G(A) 1 , 
which consists of all cuspidal automorphic functions on G(A) 1 and is 
embedded into L^ sc ( G(F)\g (A) 1 ). 

As in Section 1.3 of [Arl2j . denote by A cusp (G), .4.2 (GO, and A{G) re- 
spectively, the set of irreducible unitary representations of G(A) whose 
restriction to G(A) 1 are irreducible constituents of L^ usp (G'(F)\G'(A) 1 ), 
L2 isc (G(F)\G(A) 1 ), and L 2 (G(F)\G'(A) 1 ), respectively. It is clear that 

(2.3) Ausp(G) C A 2 {G) c A(G). 

In particular, when G = Rf'/f(N) = R^//^(GLat), the Weil restriction, 
the notation is simplified as follows: .A CUS p(iV) := A C us P (G), A2 (N) : = 
A 2 (G), and A(N) := A(G). 

2.1. Discrete spectrum of classical groups. Recall more notation 
from Sections 1.3 and 1.4 of |Arl2] and Section 2.3 of [Mkl2] in order to 
state Arthur's theorem in more explicit way. A^ usp (G) and A^iG) are 
defined in the same way as *4. C usp(G) and A 2 (G), respectively, except 
that the irreducible representations 7r of G(A) may not be unitary. The 
set A + (N) can be easily defined in terms of the set A(N). The elements 
in A(N) can be explicitly given as follows. 

Let N = YH=i Ni be a any partition of N and P = MpNp be the 
corresponding standard (upper-triangular) parabolic subgroup of GL^. 
For each i, take 7Tj G A2 (Ni) and form an induced representation 

(2.4) 7T := Z p (tti g> 7T 2 g> • • • <g> vr r ), 

which is irreducible and unitary. The automorphic realization of 7r 
is given by the meromorphic continuation of the Eisenstein series as- 
sociated to the induced representation. The set A(N) consists of all 
elements of type (12.41) . Then the set A + (N) consists of all elements of 
type as in (12 .4p . but with 7Tj belonging to A^Ni) for i = 1, 2, • • • , r. 
Hence one has 



The corresponding sets of global Arthur parameters are given by 



(2.6) ^ cusp (iV) C tt 2 (JV) = ^sim(iV) c * e ii(A) C *(A7) C V + (N). 



More precisely, the set ^ 2 (N) = ^ S i m (N) consists of all pairs (r, b) with 
N = ab (a,b > 1) and r G A cnsp (a). The subset ^ cusp (N) consists of 
all pairs (r, 1), i.e. with b = 1 and N = a and r G *A C usp(^0- The set 
ty(N) consists of all possible formal (isobaric) sum 



(2.5) 



Aus P (A0 C A 2 (N) c A(N) c A + (N). 



(2.7) 



l/j := ffl V>2 ffl • • • ffl 1p; 
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where ipi belongs to \I/ S i m (Aj) with any partition iV = YH=i^i an d 
hence ipi = fa, fej) with A^ = afci and r« G *4. C usp( a i)- The elements 
of tf+(JV) are of the form (J2ZD with r ; e ./^(a*). The set *eii(A0 
consists of all elements of the form (12. 7p with the property that ipi and 
ipj are not equal if i ^ j. The equality of ipi and ipj means that &j = 6j 
and Tj = Tj. 

For ^ G *$?(N) as given in (12.71) . define the dual of ip by 

(2.8) ip* :=ipimipzm---mip* } 

where ip* := (r*, 6j) with r* being the dual ijf) y of the Galois twist rf 
of Tj, for i = 1, 2, • • • , r. Denote by ^f(N) the subset of ^f(N) consisting 
of all conjugate self-dual parameters, i.e. ip* = ip. Hence one has 

(2.9) ^cusp(iV) := 5(iV)n^ cusp (iV), 

(2.10) $sim(iV) := 5(iV)n^ sim (iV), 

(2.11) *eii(iV) := *(iV)n# e ii(A0- 

By Theorem 1.4.1 of [Arl2j and Theorem 2.4.2 of |Mkl2j . for each con- 
jugate self-dual, simple, generic global Arthur parameter ip in ^ cnsp (N), 
there is a unique (up to isomorphism class of endoscopy data) en- 
doscopy group G^p in 8 e u(N), the set of twisted elliptic endoscopy 
groups of Kf'/f(N), with the weak Langlands functorial transfer from 
G,p to Rf'/f(N). Note that is a simple algebraic group defined over 
F which is either an F-quasisplit unitary group, a symplectic group or 
an F-quasisplit orthogonal group. The set of all these simple classical 
groups is denoted by 8 S i m (N). The theme of [Arl2j and [Mkl2] is to 
classify the discrete spectrum for these classical groups G in £ S i m (N). 

Note that when F' = E and G^ = Ue/f(N), the endoscopy data 
contains the L-embedding £ Xk from the L-group L G^, to the L-group 
of Re/f(N), which depends on the sign k = ±. The two embeddings 
are not equivalent. This will be more specifically discussed in Section 
where the endoscopy groups of \Je/f(N) are discussed. The general 
theory of twisted endoscopy is referred to [KtS99j. 

For each G G £ S i m (N), denote by ^(G) the set of global conju- 
gate self-dual Arthur parameters ip in ^(N), which factor through the 
Langlands dual group L G in the sense of Arthur (Page 31 in [Arl2j and 
also [Mkl2j ). When ip belongs to \l/ e ii(AQ, the above property defines 
\I/ 2 (G). Similarly, for each G G £ e ii(A0 (the twisted elliptic endoscopy 
groups of Rf'/f(N)), one defines the set of global Arthur parameters 
* 2 (G). As remarked in Pages 31-32 of |A7T2] (and also in [Mkl2j ) the 
elements of ^(G) may be more precisely written as a pair (ip,ipa) and 
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the projection 

(V>, V>g) ^ ^ 

is not injective in general. However, as explained in Page 33 of |Arl2] 
(and also in |Mkl2j ). for ip G ^ e n(N), there exists only one G G £ e \\(N) 
and one ipG G ^2 (GO projects to '0, and hence the set ^(G) can be 
viewed as a subset of \l/ e ii(iV). Moreover, one has the following disjoint 
union: 

^ell(iV) = U Gegcn(N ^ 2 (G). 

Note that the set ^(G) is identified as a subset of \& e u(iV) via the 
L-embedding £ Xk in the endoscopy data (G, £ Xf J. 

Theorem 1.5.2 of [A7I2] and Theorem 2.5.2 of [Mkl2j give the fol- 
lowing decomposition of the discrete spectrum of G G S S i m (N), which 
is a refinement of Theorem ll.lt 

(2.12) Ll sc (G(F)\G(A)) - ®^ 9m ©. e n, M m**, 

where linear character defined as in Theorem 1.5.2 of |Arl2] in 

terms of ip canonically, and H^(e^) is the subset of the global Arthur 
packet 11^ consisting of all elements ir = ® v t: v whose characters are 
equal to e^, and finally, is the multiplicity which is either 1 or 2, 
depending only on the global Arthur parameter ip. See Theorem 1.5.2 
of [A7T2] and Theorem 2.5.2 of [Mkl2] for details.^ 

Note that in the case of unitary groups, the set ^(G) depends on a 
given embedding £ Xk of L-groups, as described in [Mkl2| Section 2.1]. 
Hence one should use the notation ^(G, £ Xk ) for ty 2 (G). If £ Xk is fixed 
in a discussion, it may be dropped from the notation if there is no 
confusion. 

It is clear that the subset n^(e^) of the global Arthur packet 
consists of all members in which occur in the discrete spectrum of 
G. It is discussed in [Arllj and |Arl2| Section 4.3] that 11^ (e^) is in 
fact the subset of |Arl2l Section 4.3] consisting of all members which 
may occur in the whole automorphic L 2 -spectrum of G. As remarked 
in [Arl2l Section 4.3], it is a long term study problem to understand 
other possible automorphic representations in the given global Arthur 
packets |Arl2l Section 4.3]. In this paper, for simplicity, the set U^(e^) 
is called the automorphic L 2 -packet attached to the global Arthur 
parameter ip. 
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2.2. Arthur-Langlands transfers. For each G G £ S i m (A), by the 
definition of twisted elliptic (simple type) endoscopy data, there is L- 
homomorphism 

= Cv> : L G -» l Rf'/f(N), 

where Rp//p(A) = Rf'/f(GL n ). The corresponding Langlands func- 
torial transfer from G to Rp//p(A) is given as follows. 

Take ip G ^(G). The theory of stable trace formula ( [Arl2] ) pro- 
duces the automorphic L 2 -packet n^(e^), which consists of members 
in the global Arthur packet IT^ that belong to .4.2 (GO . On the other 
hand, if) can be expressed as in (12. 7\i 

where ipi belongs to ^ B i m (Ni) with a partition N = 5^I=i ^ an d hence 
■0i = (Tj, 6j) with Aj = dibi and 7} G 4. C usp(a-j) being conjugate self-dual. 
For each ipi G \l/ S i m (A^), by the theorem of Moeglin and Waldspurger 
QMW89J and Theorem 1.3.3 of |Arl2j ). there is a unique (up to isomor- 
phism) irreducible residual representation 7Tj of Rp//p(Aj)(A) which is 
generated by the Speh residues of the Eisenstein series with cuspidal 
support 

(R^Krvro, 

and is sometimes denoted by A(rj,6j). Then one defines the Arthur 
representation of Rp//p(A)(A), 

(2.13) 7ty := l P (ni <g> 7r 2 <g> • • • <g> n r ), 

which is induced from the parabolic subgroup of Rp//p(A) attached 
to the partition A = YH=i ^i- ^ * s an irreducible unitary automor- 
phic representation of Rp//p(A)(A), whose automorphic realization is 
given by the Langlands theory on meromorphic continuation of residual 
Eisenstein series on Rpyp(A)(A). 

The Arthur-Langlands transfer is a Langlands functorial transfer 
corresponding to which takes the set LT^(e^) to the automorphic 
representation 7ty, in the sense that for each 7r G Il^(e^), the Satake 
parameter at each unramified local place v, c(tt v ), matches the corre- 
sponding Satake parameter c(ir^ tV ), i.e. 

^( c W) = c (^.«)- 

This gives the functorial interpretation of the Arthur decomposition 
(12~T2|) for the discrete spectrum L 2 disc (G(F)\G(A)) for each G G £ sim (A), 
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which can be presented by the following diagram: 

* a (G) 

if, 

s \ 

A 2 (G) n^(e^) — > vr^ .4(jV) 

Remark 2.1. zs an important problem to study when the Arthur- 
Langlands transfer from any member it in the automorphic L 2 -packet 
n^(e^) to the Arthur representation tt^ is compatible with the local 
Langlands functorial transfers at all local places. In particular, when 
the global Arthur parameter ip 6 ^(G) is not generic, it is in gen- 
eral to expect that the image under the Arthur- Langlands transfer of all 
members in Il^(e^) contains automorphic representations other than 
tt^. It is an even more subtle problem to figure out what automorphic 
representations should be expected in the transfer image of the automor- 
phic L 2 -packet n^(e^) for a given non-generic global Arthur parameter 

i> e * 2 (G)- 

On the other hand, it is also important to figure out what kind auto- 
morphic representations are expected to be in an automorphic L 2 -packet 
n^(e^) for a given global Arthur parameter ip 6 ty 2 {G). Moeglin pro- 
vides conditions and conjectures in [Mgll| on whether an automorphic 
L 2 -packet EL^(e^) contains nonzero residual representations, while the 
thesis of Paniagua- Taboada in [PTllj provides cases that an automor- 
phic L 2 -packet n^(e^) may contains no cuspidal members for SC>2n- 
The work of Liu and Zhang joint with the author in |JLZ12] constructs 
residual representations in certain automorphic L? -packets IT^(e^) for 
F-quasisplit classical groups. 

3. Endoscopy Structures for Classical Groups 

For each (G,£ Xk ) G £ S im(N) and for each ip e ^f 2 (G), the decompo- 
sition as in (12.71) 

V> := fa ffl 1P2 ffl • • • ffl A 

also implies an endoscopy structure for ip and hence for the automor- 
phic representations in n^(e^). 
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Recall that ipi belongs to ^ S im_{Ni) with the partition N = Ni 
and that ipi = (7*, bi) with the property that Ni = afti and 7* G 
vA C usp(ai) is conjugate self-dual, i.e. r* = (rf) v = Tj. For each -0j G 
^sim(-/Vj), there is a unique twisted elliptic (simple type) endoscopy 
group Gi (as part of the endoscopy datum) in £ sim (iVj). Hence for ip, 
one may consider 

Gi x G 2 x • • ■ x G r . 

One may call this product of groups an (generalized) elliptic endoscopy 
group of G determined by ip. 

When F' = E and G = XJ(N) = V E/F (N), one should add the 
L-embedding 

: L (U(iV0 x • • • x XJ(N r )) -> L XJ(N) 

as given in |Mkl2t (2.1.13)], and the sign k = • • • , K r ) is defined 
as in [Mkl2l (2.1.12)] with m = (-1) N - N * for % = 1, 2, ■ ■ ■ , r. The L- 
embeddings from the two L-groups to the L-group l Re/f(N) are given 
as in |Mkl2l (2.1.14)]. 

The main constructions of endoscopy correspondences considered in 
this paper are to take the following decomposition of ip: 

(3.1) V = M)fflVi\r-a6 

with r G ^4 CU sp(a) being conjugate self-dual, and t^N-ab £ ^c\\(N — ab). 
Note that (r, b) G ^/ sim (ab). Take G G £ sim (a&) and if G £ sim (iV — a6) 
and consider the endoscopy structure 

(3.2) C7 x H ->■ C7, 

with (r, 6) G ^2(^0) and ipN-ab G ^2(H). The explicit constructions 
for the endoscopy transfers associated to (13. ip will be discussed below 
case by case and are expected to confirm Conjecture [L2] for those cases. 
For r G ~4 C usp(a), one may use the notation that a = a T to indicate 
the relation between Rp>/p(a) and r. The explicit constructions for the 
endoscopy correspondences are distinguished according to the types of 
the functorial transfer of r and the parity of a T , which will be discussed 
with details below. 

When F' = F, a self-dual r G A cusp (a) is called of symplectic type if 
the (partial) exterior square L-function L s (s, r, A 2 ) has a (simple) pole 
at s = 1; otherwise, r is called 0/ orthogonal type. In the latter case, 
the (partial) symmetric square L-function L s (s, r, sym 2 ) has a (simple) 
pole at s = 1. 

When F' = E, the quadratic field extension of F, a conjugate self- 
dual r G *4 C usp( a ) yields a simple generic parameter (r, 1) in ^(o) 
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which produces a sign n a for the endoscopy data (Ue/j?(a), £, XKa )- By 
Theorem 2.5.4 of [Mkl2] . the (partial) L-function 

L(s, (r, l),Asai"^) 

has a (simple) pole at s — 1 with the sign ^( T ,i) = K a ■ (— l) a_1 (see 
also pGEH Theorem 8.1] and |Mkl2L Lemma 2.2.1]). Then the irre- 
ducible cuspidal automorphic representation r or equivalently the sim- 
ple generic parameter (r, 1) is called conjugate orthogonal if i]( T ,i) = 1 
and conjugate symplectic if f](r.i) — ~ 1> following the terminology of 
[GGP12L Section 3] and [MEI21 Section 2]. Here L s {s, (r, 1), Asai + ) is 
the (partial) Asai L-function of r and L s (s, (r, 1), Asai - ) is the (par- 
tial) Asai L-function of t®uje/f, where ue/f is the quadratic character 
associated to E/F by the global classfield theory. 

The sign of (r, b) G ^(afr) can be calculated following |Mkl2l Section 
2.4]. Fix the sign n a as before for the endoscopy data (U.E/jr(a),£ x ), 
the sign of (r, 1) is 

V(r,i) =Vt = K a (-l) a ~ l . 
Hence the sign of (r, b) is given by 

77 (Tj6) = ^(-l)- 1 -*- 1 = ^ a (-l) a+6 = ^(-l) 6 - 1 . 

As in [Mkl2l (2.4.9)], define 

Ka6 := «: a (-l) a6 - a - 6+1 . 

Then K ab (-l) ab - 1 = ^(-l)^ 1 = 7/ (T)6) and hence = ^(-l)^" 1 ) 6 , 
which gives the endoscopy data (UE/F(ab), £ Kab )- 

3.1. Endoscopy structure for S0 2n+ i. Assume that G G S S i m {N) 
is a L-rational form of S0 2n +i. Then G must be the L-split S0 2n +i, 
N = 2n and G v = Sp 2n (C). 

In this case, the Langlands dual group L G is a direct product of the 
complex dual group G y and the Galois group IV = Gal(L/L). For 
each if) G ^2(S0 2n +i), which is of form 

(3.3) if) = (r, b) EH ip 2n -ab, 

with r G ^4 C us P ( a ) being self-dual and b > 0, such that (r, 6) is of 
symplectic type, and with ip2n-ab being a (2n — a6)-dimensional elliptic 
global Arthur parameter of symplectic type, there are following two 
cases, which have to be distinguished when constructing the endoscopy 
correspondences by integral transforms. 

Case (a=2e): When a = a T = 2e, if r is of symplectic type, then 
b = 21 + 1; and if r is of orthogonal type, then b = 21. The endoscopy 
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group of S0 2n+ i associated to the global elliptic Arthur parameter ip 
as in (I3.3p is 

(3.4) SCU + i(A) x S0 2n _ a6+1 (A) -> S0 2n+1 (A). 

Following the formulation of Wen- Wei Li ( |Liwllj ). one can have the 
following variants: 

S(W(A) x s5 2n _ o6 (A) -> s5 2n (A), 

and 

(A) Sp 2n (A). 

Case (a=2e+l): When a = a T = 2e + 1, then r is of orthogonal 
type and b = 21. The endoscopy group of S0 2n+ i associated to the 
global elliptic Arthur parameter ip as in (13. 3p is also 

(3.5) SO Qb+1 (A) x S0 2n _ a6+1 (A) -> S0 2n+1 (A). 

In fact, the explicit constructions of the integral transforms for this case 
use the following endoscopy group of S0 2n+ i associated to the global 
elliptic Arthur parameter ip: 

(3.6) Sp ab (A) x S^ 2n _ a6 (A) S0 2n+1 (A). 

Note that this endoscopy structure is not given from |Arl2j , but is given 
in [Liwllj . By the duality in the sense of Howe, one may consider in 
this case the endoscopy group of Sp 2n (A): 

(3.7) SO a6+1 (A) x S0 2 „_ ab+1 (A) -> §5 2n (A). 

Hence the constructions discussed for those cases would be natural ex- 
tensions of the classical theta correspondences for reductive dual pairs. 
More details will be given in Section 7. 

3.2. Endoscopy structure for Sp 2n . Assume that G G £ S i m (iV) is 
equal to Sp 2n . Then iV = 2n + l and G v = S0 2n+ i(C). In this case, the 
Langlands dual group L G is a direct product of the complex dual group 
G y and the Galois group T F = Gal(F/F). For each ip e ^ 2 (Sp 2n ), 
which is of form 

(3.8) V = (r, b) ffl ^2n+l-ab, 

with r G *A C us P ( a ) being self-dual and b > 0, such that (r, b) is of 
orthogonal type and with ip2n+i-ab is a (2n — a6+l)-dimensional elliptic 
global Arthur parameter of orthogonal type, there are following two 
cases, which have to be distinguished when constructing the endoscopy 
correspondences by integral transforms. 

Case (a=2e): Assume that (r, b) is of orthogonal type with even 
dimension. In this case, one must have that a = a T = 2e. If r is of 
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symplectic type, then 6 = 2/ must be even, and if r is of orthogonal 
type, then b = 21 + 1. The endoscopy group of Sp 2n associated to the 
global elliptic Arthur parameter ip as in (13. 8p is 

(3.9) SO ah (A) x Sp 2n _ o6 (A) Sp 2n (A). 

Note that the orthogonal group SO a f, may not be F-split. 

Case (a=2e+l): Assume that (r, b) is of orthogonal type with odd 
dimension. In this case, one must have that a = a T — 2e + 1, r is of 
orthogonal type and b = 21 + 1. The endoscopy group of Sp 2n associated 
to the global elliptic Arthur parameter ip as in (13 .8p is 

(3.10) S Po6 _ 1 (A) x S0 2n _ a6+1 (A) Sp 2n (A). 
Again the orthogonal group S0 2n -ab+i m &y not be F-split. 

3.3. Endoscopy structure for S0 2n . Assume that G e £ S i m (N) is an 
F-quasisplit even special orthogonal group SO 2n (gv0, which is denoted 
by S0 2n {v) i n |A7l2l . Then N = 2n and G v = S0 2n (C). 

Following |Arllt Section 1.2], for G = S02 n (^), the set of elliptic en- 
doscopy groups of G is parameterized by pairs of even integers (N[, iV 2 ) 
with < N[ < N' 2 and N = N[ + N' 2 and by pairs (rj[, r)' 2 ) of quadratic 
characters of the Galois group T F with r] V = r}' x r}' 2 . The corresponding 
elliptic endoscopy groups are F-quasisplit groups 

(3.11) G" = SO^K)xSO^(r7 2 ), 
with dual groups 

G N = SOjv((C) x SOat.(C) c SOat(C) = G v . 

The group G = SOat^) has also twisted elliptic endoscopy groups, 
which are parameterized by pairs of odd integers (iV{,iV 2 ) with < 
N[ < N 2 and = N[ + N 2 , and by pairs of quadratic characters 
(r?i,T?2) °f ^e Galois group T F with r\ = rj[ ■ rj 2 . The corresponding 
twisted elliptic endoscopy groups are 

(3.12) G' = Sp^,_ 1 x Spj^,_ 1 
with dual groups 

G' v = SO^,(C) x SO^,(C) c SOjv(C) = G v . 

For the purpose of this paper, the above general structures will be 
specialized as follows. For each ip e ^2(S0 2n ), which is of form 

(3.13) V = (t, b) ffl ^j 2n _ ab , 

with r G ^.cusp(a) being self-dual and b > such that (r, b) is of 
orthogonal type, and with ip2n-ab is a {2n — a6)-dimensional elliptic 
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global Arthur parameter of orthogonal type, there are following two 
cases, which have to be distinguished when constructing the endoscopy 
correspondences by integral transforms. 

Case (a=2e): Assume that (r, b) is of orthogonal type with even 
dimension. In this case, one must have that a = a T = 2e. If r is of 
symplectic type, then b = 21 must be even; and if r is of orthogonal 
type, then b = 21 + 1. The standard elliptic endoscopy group of S0 2n 
associated to the global elliptic Arthur parameter ip as in (I3.13P is 

(3.14) SCU(A) x S0 2n _ a6 (A) -> S0 2n (A). 

Note that the orthogonal groups here may not be F-split. 

Case (a=2e+l): Assume that (r, b) is of orthogonal type with odd 
dimension. In this case, one must have that a — a T — 2e + 1, r is of 
orthogonal type and b = 21 + 1. The twisted elliptic endoscopy group of 
S0 2n associated to the global elliptic Arthur parameter ip as in (13.131) 
is 

Sp a6 _ 1 (A) x Sp^^A) S0 2n (A). 
Note that the orthogonal groups here may not be F-split. 

3.4. Endoscopy structure for \Je/f{N). The (standard) elliptic en- 
doscopy groups of the F-quasisplit unitary groups \Je/f(N) = U(iV) 



are determined by J. Rogawski in Rg90j Section 4.6]. The discussion 



here follows from [Mkl2t Section 2] for consistence of notation. 

Assume that U(iV) be the F-quasisplit unitary group defined by an 
non-degenerate hermitian form qy of N variables associated to the qua- 
dratic field extension E/F. As a (twisted) endoscopy data of Re/f(N), 
one needs to keep the sign k in (U(AQ, £ Xk ). 

The set of (standard) elliptic endoscopy groups (or data) of U(iV) 
are given as in [M kl2l Section 2.4] and also in [JZ13] by 

(3.15) (U(JV0 x U(N 2 ),CxJ = (U(iVi) x U{N 2 ),k) 

with N!,N^> and N = Ni + N 2 ; and k = ((-1) N ~ N \ (-1) N ~ N2 ). 
Write i[) e \l>2(U(^ r )) in form 

(3.16) V = (r, b) ffl i) N ~ a \ 

with t G A cnsp (a) being conjugate self-dual and b > 0; and (r, b) G 
* 2 (U(a6)), and ip N ~ ab G vj> 2 (U(iV - ab)). In order for (r, b) to be in 
\l/ 2 (U(a&)), the sign T]^) of the global simple Arthur parameter (r, b) 
has the property that 

V(r, b ) = ^(-l)" 6 - 1 = ^(-l) 6 " 1 = K(-1) N ~, 

which implies that r\ T = n(—l) N ~ b and leads to the following cases: 
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(1) Case (N=2n): If r or (r, 1) is of conjugate orthogonal type, 
then 7] T = K a (— l) a_1 = 1 and hence one must have that (— l) b = 
k; and if r or (r, 1) is of conjugate symplectic type, then r\ T = — 1 
and hence one must have that (— l) b = —k. 

(2) Case (N=2n+1): If r or (r, 1) is of conjugate orthogonal type, 
then i] T = 1 and hence one must have that (— l) b = —k; and if 
r or (r, 1) is of conjugate symplectic type, then r\ T = — 1 and 
hence one must have that (— l) 6 = k. 

The standard elliptic endoscopy group of LV associated to the global 
elliptic Arthur parameter ip as in ( 13.16)) is 

(3.17) (U(afo) x U(N - ab), ((-l) afc , (-l) a6 )) 
UN— 2n, and 

(3.18) (U(a6) x U(iV - aft), ((-l) a6+1 , (-l) a6 )) 
if N = 2n + 1. 

4. Fourier Coefficients and Nilpotent Orbits 

In order to formulate the automorphic integral transforms which pro- 
duce various types of endoscopy correspondences, it is the key step to 
define the right kernel functions for a given setting of the endoscopy cor- 
respondences. In principle, the kernel functions are defined by means of 
discrete series automorphic representations of simple type in the sense 
of Arthur and by means of the structures of Fourier coefficients of au- 
tomorphic forms defined in terms of nilpotent orbits. The goal of this 
section is to state and discuss a general conjecture on the structure of 
Fourier coefficients of discrete series automorphic representations and 
their relations to Arthur classification of the discrete spectrum, which 
can be viewed as a natural extension of the global Shahidi conjecture 
on the genericity of tempered L-packets. 

4.1. Nilpotent orbits. Let G G £ S i m (N) be an F-quasisplit classical 
group and g be the Lie algebra of the algebraic group G. Let jV(fl) be 
the set of all nilpotent elements of g which is an algebraic conic variety 
defined over F and is called the nilcone of q. Under the adjoint action 
of G, which is defined over F, the nilcone A/"(fl) is an algebraic G- variety 
over F. Over algebraic closure F, the F-points N(q)(F) decomposes 
into finitely many adjoint G(F)-orbits O. It is a well-known theorem 
(Chapter 5 of |CM93j ) that the set of those finitely many nilpotent 
orbits are in ono-to-one correspondence with the set of partitions of 
Nq with certain parity constraints (called G-partitions of No), where 



AUTOMORPHIC INTEGRAL TRANSFORMS FOR CLASSICAL GROUPS I 21 



N G = N + 1 if G is of type S0 2 „+i, N G = TV - 1 if G is of type Sp 2n , 
and Nq = N if G is of type S0 2n or a unitary group Ujv = U(iV). 

4.2. Fourier coefficients of automorphic forms. Over F, the F- 
rational points jV(g)(F) decomposes into F-stable adjoint G(F)-orbits 
O st , which are parameterized by the corresponding G-partitions of Nq- 
For X G A/"(g)(F), the theory of the sl 2 -triple (over F) asserts that 
there is a one-parameter semi-simple subalgebra H(t) in g with t G F 
and a nilpotent element Y G A/"(g)(F) such that the set {X,H(1),Y} 
generates a F-Lie subalgebra of g, which is isomorphic to sl 2 (F). This 
means that 



Under the adjoint action of the Lie algebra 0(F) decomposes into 
a direct sum of adjoint ^(l)-eigenspaces 

(4.1) = fl-m © * * * © 0-2 © 0-1 © 00 © 01 © 02 © ••■ 0m 

for some positive integral m, where g r := {A G | [ft(l),A] = rA}. 
Then X G 2 and K G 0_ 2 . Let L be the algebraic F-subgroup 
of G such that the Lie algebra of L is 0o- Then the adjoint G(F)- 
orbits in the stable orbit 0% of X are parameterized by the adjoint 
Lo(F)-orbits of maximal dimension in either 2 (F) or 0_ 2 (F). For F- 
quasisplit classical groups, the classification of the adjoint G(F)-orbits 
in the stable orbit of any X G A/"(g)(F) is explicitly given by J.-L. 
Waldspurger in |W01l Chapitre I]. 

Let Vx be the unipotent subgroup of G whose Lie algebra is ©«> 2 0i. 
There is a canonical way to define a nontrivial automorphic character 
tpx on Vx(A), which is trivial on Vx{F), from a nontrivial automorphic 
character if)p of F\A. More precisely, it is given by: for any v G Vx(A), 



Let it belong to A(G). The Fourier coefficient of <p n in the space of 
7r is defined by 



Since <p n is automorphic, the nonvanishing of J-"^ x ((p w ) depends only 
on the G(F)-adjoint orbit O x of X. 

Define n(v? vr ) := {X G Af(g) \ J I "^ x ((p 7T ) ^ 0}, which is stable under 
the G(F)-adjoint action, n 171 ^^) is the union of maximal G(F)-adjoint 
orbits in n(ip n ), according to the the partial ordering of the correspond- 
ing stable nilpotent orbits, i.e. that of corresponding partitions. Define 
simply that ti(7r) to be the union of n"^^) for all ip^ running in the 



[X,Y] = H(1), [h{l),X] = 2X, [h{l),Y] = -2Y. 



4>x(v) :=^(tr(y-ln(v))). 
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space of tt and define n m (7r) to be the union of maximal G(F)-adjoint 
orbits in n(7r), in the same way as n m ((p). 

4.3. Example of GL^. When G = GL^, every G(F)-orbit is stable 
and the set of all nilpotent orbits of GLn(F) in JV(q)(F) are param- 
eterized by partitions of N. The well known theorem proved by J. 
Shalika and I. Piatetski-Shapiro, independently, says that every tt in 
"4 cusp (iV) is generic, that is, there is an automorphic form ip^ in the 
space of tt having a nonzero Whittaker- Fourier coefficient. Following 
the definition of Fourier coefficients above, the Whittaker-Fourier coef- 
ficients are attached to the regular nilpotent orbit and hence have the 
partition [N] of the integer N. Therefore, one has that n m (7r) = {[A]} 
for any tt G A CUBp (N). 

What happens if tt belongs to A%(N) in general? As discussed in 
Section 2.2, if tt belongs to A2(N), by a theorem of Moeglin and Wald- 
spurger, tt is of form A(r, b), with r G *4. C us P (a) and A = ab. The main 
theorem in [JE12] shows that n m (A(r,6)) = {[a b ]}. See also [G06] for 
a sketch of different proof. 

In general, for tt G A(N), the result is described in the following 
conjecture. 

Conjecture 4.1. Let tt G *<4(A) be of form 

TT^ := l P (lTi <g> 7T 2 ® ■ ■ ■ (g) 7T r ), 

with N = YTi=iNi and with Ni = afci, iii = A(rj,6j) G A2{Ni), and 
n G Acwp( a i) for i = 1,2, - ■■ ,r. Then 

n m (n) = [a b ^} + [a^} + --- + [a b ;}. 

Note that this conjecture can be considered as a special and more 
precise version of a more general conjecture discussed by Ginzburg in 
[G06J and could be verified by extending the argument of [JL12j . 

4.4. Fourier coefficients and Arthur parameters. In this section, 
after an explanation of Conjecture HJ] in terms of Arthur parametriza- 
tion, a general conjecture will be stated on relation between Fourier 
coefficients and Arthur parameters for classical groups. 

For 7r G A(N) of form, tt^ := Xp(7ri®7r 2 (g)- ■ -®7r r ), with N = J2l=i ^ 
and with Aj = aA, ^ = A^,^) G ^(A^), and r { G A cusp (ai) for 
i — 1, 2, • • • , r, as in Conjecture 14.11 the global Arthur parameter ip 
attached to tt belongs to ^>(N) and has the form 

^ = (r 1 ,6i)ffl(r 2 ,6 2 )ffl---ffl(r r ,6 r ) 

with (rj, bi) G ^/ 2 (aj6j) for i = 1, 2, • • • , r. It is not hard to figure out 
that as a representation in the a^-dimensional complex vector space, 
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the irreducible representation (r,;, hi) should correspond to the partition 
[b^] of the integer for i = 1, 2, • • • , r, and hence the partition of N 
corresponds to the direct sum of (tl, bi), (r 2 , b 2 ), • • ■ , {j r , b r ) should be 



Note that the partition [a^ 1 ] + [a b 2 2 ] + • ■ ■ + [a b r r ] in Conjecture HJ] 
is the image of the partition [b^b^ 2 ■ ■ ■ b^ r ] under the Barbasch-Vogan 
duality mapping r] g v g from the dual Lie algebra g v of g to g with g = 
gl N ([B V85j and |Ac03j ). This suggests a conjecture for F-quasisplit 
classical groups, which is formulated in terms of global Arthur packets. 

Conjecture 4.2 (Upper Bound of Maximal Fourier Coefficients). For 
a F-quasisplit classical group G, let LT^(e^) be the automorphic L 2 - 
packet attached to a global Arthur parameter if) G ^ 2 {G). Assume that 
p{ip) is the partition attached to (if), G V (C)). Then for any tt G Il^(e^), 
n m (n) is bounded above by the partition Vg v ,s(p(' 1 P)) ■ 

It remains very interesting to figure out exactly what n m (n) are for 
7r G n^(e^). 

When the global Arthur parameter ip G ^(G) is generic, the par- 
tition p(ip) attached to (ip, G V (C)) must be the trivial partition, i.e. 
corresponding to the zero nilpotent orbit in g v (C). Then the Barbasch- 
Vogan duality ^ v , g (p(V ; )) must be the partition corresponding to the 
regular nilpotent orbit in g(F) ([BV85] and |Ac03j ). Following the exis- 
tence of the Arthur-Langlands transfer ( [Arl2j and [Mkl2] ) and the au- 
tomorphic descent construction of Ginzburg-Rallis-Soudry ( |GRSllj ). 
there exists an irreducible generic cuspidal automorphic representation 
7r in the automorphic L 2 -packet Il^(e^), such that n m (7r) = ?7 g v !0 (p(V'))- 
By the generalized Ramanujan conjecture for the generic Arthur pa- 
rameter if}, this is a global version of the Shahidi conjecture, which 
claims that any global tempered L-packet has a generic member. 

On the other hand, consider a global Arthur parameter of simple 
type, ip = (r,b) G ^(G), with r G ^4 C us P (a) being conjugate self-dual. 

If take G = S02 n +i, then In = ab. Hence r is of orthogonal type if 
and only if b = 21; and r is of symplectic type if and only if b = 21 + 1. 
The partition attached to if) is p(ip) = [b a ], and the Barbasch-Vogan 
dual is 



If take G = S02 n , then 2n = ab. Hence r is of orthogonal type if and 
only if b = 21 + 1; and r is of symplectic type if and only if b = 21. The 
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partition attached to if) is p(if>) = [b a ], and the Barbasch-Vogan dual is 

J>] if 6 = 2Z; 

^ 8 (EW) = | K - 1(a _ 1)1] if 6 = 2/ + 1 . 

If take G = Sp 2n , then 2n + 1 = afe. Hence r must be of orthogonal 
type and b = 21 + 1 and a = 2e + 1. The partition attached to if> is 
p(V>) = [& a ], and the Barbasch-Vogan dual is 

v>W) = K _1 («-i)]- 

Following [G03] . [G06j . [G08], [GRSTT] . jJLT2] and |JLZ12j . Liu and 

the author are able to show in |JL13j that for any simple global Arthur 
parameter ip for all F-split classical groups, Conjecture 14.21 holds and 
there is a member n e Il^(e^) such that n m (7r) = T) g v >g (p{if))). 

These two extreme cases may suggest the technical complication on 
how to figure out what n m (n) are for n 6 (e^) or even the upper 
bound partition and the lower bound partition for the automorphic 
L 2 -packet n^,(e^). Some testing cases are considered in [JL13j . which 
support that there should always be a member ir e n^(e^) such that 
n m (7r) = 7) B v )g (p(if))) in general. 

Conjecture 4.3 (Maximal Fourier Coefficients). For a F-quasisplit 
classical group G, let Il^(e^) be the automorphic L 2 -packet attached 
to a global Arthur parameter ip G ^(G). Assume that pijp) is the 
partition attached to (ip, G V (C)). Then there exists at least one member 
it G n^(e^), such that n m (7r) is equal to the partition ^ g v , g (p(V ; )); ^- e - 
the upper bound of the maximal Fourier coefficient in Conjecture \4-S\ 
can be achieved by members in n^(e^) 

For a non-generic global Arthur parameter if) e \1/ 2 (G), it is an inter- 
esting problem to determine automorphic representations occurring as 
members in the automorphic L 2 -packet Il^(e^) by means of the basic 
invariants attached to automorphic forms, and this problem is closely 
related to the topic on CAP automorphic representations, which was 
initiated by Piatetski-Shapiro through his work on Saito-Kurokawa lift- 

ing (IESS31). 

Along this line of ideas, it is natural to take the structure of Fourier 
coefficients of irreducible cuspidal automorphic representations as basic 
invariants and study the classification problem. The approach should 
go back to the root of the theory of automorphic forms. However, the 
representation-theoretic approach towards this problem goes back to 
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the notion of the rank of automorphic forms or automorphic represen- 
tations, introduced by R. Howe ( |Hw81j and |Hw 82j). The work of 
J.-S. Li ( |Li95j and |Li97j ) give a complete characterization of singu- 
lar automorphic representations of classical groups in terms of theta 
correspondences. From the point of view of the Arthur classification 
of discrete spectrum, Moeglin published a series of papers on classify- 
ing automorphic representations which are expected to have quadratic 
unipotent Arthur parameters (|M g91| and |Mg94] ) , which have deep 
impact to the understanding of automorphic forms related to the work 
of Kudla and Rallis on their regularized Siegel-Weil formula ([KR94J) 
and the further arithmetic applications of Kudla ( |Kd97j ). A family of 
interesting examples of cubic unipotent cuspidal automorphic represen- 
tations of the exceptional group of type G2 were constructed through 
the exceptional theta correspondences by Gan, Gurevich and the au- 
thor in [GGJ02J, which produces examples of cuspidal automorphic 
representations with multiplicity as high as one wishes. 

As one will see in late sections, the (r, 6)-theory of automorphic forms 
and the constructions of endoscopy correspondences take the structures 
of Fourier coefficients as a basic invariants of automorphic forms. With 
the Arthur classification theory in hand, it is natural to try an under- 
standing of the structure of Fourier coefficients of cuspidal automorphic 
representations in terms of the global Arthur parameters. The readers 
will see this point in the discussion in the rest of this paper and also 
in |GRS03j . [G03j . [G06] . [G08] . jJnTT] . [JL12] . and jQE]. As a side 
remark, the structure of Fourier coefficients of certain residual represen- 
tations made important progress towards the global Gan-Gross-Prasad 
conjecture ( |GGP12] ) through the a series of work of Ginzburg, Rallis 
and the author ([ GJR04] , |GJR0 5j. and |GJR09j ). Some connections 
of the Fourier coefficients to arithmetic and even mathematical physics 
can be found in [JR97j . |GGS02j . [MS12] . [GMRVlOj and jGMVllj . 

Finally, it is important to mention that the local theory goes back 
to the work of Howe on the notion of the rank for representations 
( |Hw82j ). the work of Bernstein and Zelevinsky on the notion of the 
derivatives for admissible representations of p-adic GL(n) ( |BZ77j ) and 
the work of A. Aizenbud, D. Gourevitch, and S. Sahi ( |AGS11] for 
archimedean cases, the work of Moeglin and Waldspurger on the de- 
generate Whittaker models of p-adic groups ( |MW87j ) and the work 
of Gourevitch and Sahi in [GS12] for archimedean cases, and certain 
refined structures of the Fourier coefficients was obtained by Moeglin 
in |Mg96| for p-adic tempered representations, by B. Harris in [H12] for 
real tempered representations, and by Baiying Liu and the author in 
[JL13] for cuspidal automorphic representations of symplectic groups. 
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5. Constructions of the Automorphic Kernel Functions 

The automorphic kernel functions are constructed in terms of 
automorphic forms in automorphic L 2 -packets n^ (e^ Q ), with Fourier 
coefficients associated to relatively small partitions, following the idea 
of the classical theta correspondence and the exceptional theta corre- 
spondence, which use the minimality of the kernel functions. Section 
5.1 recalls the main results of |JLZf 2] . which provide at least one con- 
cretely constructed member in these automorphic L 2 -packets II^ (e^ ). 
Section 5.2 describes explicitly a family of Fourier coefficients which 
forms a key step of the constructions. Sections 5.3 and 5.4 give explicit 
data of the constructions and conjectures on the types of endoscopy 
correspondences one expects from those constructions. The last sec- 
tion gives the construction data for unitary group case, the details of 
which will be given in a forthcoming work of Lei Zhang and the author 
( |JZ13j ). Note that the constructions discussed here essentially assume 
that d = a — 1 in the construction data. The general case (without 
this assumption) will be discussed in a forthcoming work of the author 
( |Jnl3j ). Along the way, all known cases to the conjectures will also be 
briefly discussed. 



5.1. Certain families of residual representations. Recall from 
|JLZ12] certain families of the residual representations of F-quasisplit 
classical groups, which may be considered as candidates for the con- 
structions of automorphic kernel functions. 

Following the notation in [MW95J, a standard Borel subgroup Pq = 
M N Q of G n is fixed and realized in the upper-triangular matrices. Let 
T be the maximal split torus of the center of M , which defines the 
root system R(T ,G n ) with the positive roots R + (T Q ,G n ) and the set 
Ao of simple roots corresponding to Pq. Let P = MN be a standard 
parabolic subgroup of G n (containing Pq) an d Fm be the maximal split 
torus in the center of M. The set of restricted roots is denoted by 
R(T M ,G n ). Define R + (T M ,G n ) and A M , accordingly. Furthermore, 
define Xm = X M " to be the group of all continuous homomorphisms 
from M(A) into C x , which are trivial on M(A) 1 . Then following Page 
6 of [MW95J for the explicit realization of Xm, define the real part of 
Xm, which is denoted by ReX^. 

In particular, for a partition n = r + n Q , take the standard max- 
imal parabolic subgroup P r = M r N r of G n , whose Levi subgroup 
M r is isomorphic to Rf'/fO") x G no . For any g G Rjr'/jr(r), define 
g = w r g t w r or w r c(gYw r in the unitary groups cases, where w r be the 
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anti-diagonal symmetric matrix defined inductively by 



f 

W r -1 0, 



and 



c G V E j F = {1, c}. Then any g G M r can be expressed as diag{t, h, t 1 } 
with t G RfV-fO") an d ^ ^"o- Since P r is maximal, the space of char- 
acters X^™ is one-dimensional. Using the normalization in |ShlOj . it is 
identified with C by s i— > \ s . 

Let cr be an irreducible generic cuspidal automorphic representa- 
tion of G no (A). Write r = ab. Let be an automorphic form in the 
space A(N ab (A)M ab (F)\G n (A)) A(rm(7 . Following JETS] and |MW95j . 
an Eisenstein series is defined by 



E2 b ((pA(T,l 



5cr ,s) = E((f) Am ,s) = ^2 ^<P(ig)- 

ieP ab (F)\G n (F) 



It converges absolutely for the real part of s large and has meromorphic 
continuation to the whole complex plane C. When hq = 0, a is assumed 
to disappear. 

Assume that no > here. The case when no = is similar and is 
referred to |JLZ12| Theorem 5.2]. To determine the location of possible 
poles (at Re(s) > 0) of this family of residual Eisenstein series, or 
more precisely the normalized Eisenstein series, and basic properties 
of the corresponding residual representations, one takes the expected 
normalizing factor /3 fe T (T (s) to be of the Langlands-Shahidi type, which 
is given by a product of relevant automorphic L-functions: 
(5.1) 



Pb,rA S ) '■ 

where e b ,, 
(5.2) 
and 
(5.3) 



Us 



6 + 1 



r x a) Y[ L{e bti {s) + 1, r, p) JJ Li 



eh-, 



i=l 



2s + b + 1 — 2i, and p and p are defined as follows: 



'Asai + 


if G n 


= u 2 „ 


Asai - 


if G n 


= U2n+1 


2 

sym 


if G n 


= S02n+1 


.A 2 


if G n 


= Sp 2rt or S0 2n , 


Asai~ 


if G n 


= u 2 „ 


Asai + 


if G n 


= U2„+l 


A 


if G n 


= S02n+1 


^sym 2 


if G n 


= Sp 2n or S0 2n 
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For unitary groups, Asai + denotes the Asai representation of the L- 
group of Reif{o) an d Asai - denotes the Asai representation of the 
L- group of Re/f{o) twisted by coe/f, the character associated to the 
quadratic extension E/F via the class field theory For symplectic or 
orthogonal groups, sym 2 and A 2 denote the symmetric and exterior 
second powers of the standard representation of GL (C), respectively. 
In addition, one has the following identities (Remark (3), Page 21, 
[GRSlll ): 

L(s, r x r c ) = L(s, r, p)L(s, r, p~), 

where r* = r, if F' = F; and r* = r c , if F' = E. The involution c is 
the nontrivial element in the Galois group T e /f- 

The function /3 6)Tcr (s) is used to normalize the Eisenstein series by 

( 5 - 4 ) ■ £ 2**(0A(T,6)ghT J S) := /96,T,<r(s)-ES.(0A(T,6)gXT, «)• 

In order to determine the location of the poles of i?*(0A(T,b)<g><x> s )> 
one needs to consider the following four cases: 

(1) L(s, r, p) has a pole at s — 1, and r x <r) / 0; 

(2) L(s, t, p) has a pole at s = 1, and r x a) = 0; 

(3) L(s,r, p~) has a pole at s = 1, and L(s,t x ct) has a pole at 
s = l; 

(4) L(s,r,p) has a pole at s = 1, and L(s,t x ct) is holomorphic 
at s = 1. 

The sets of possible poles according to the four cases are: 

'{6,...,^,f}, if Case (1); 

+ I {O,...,^,^ 2 }, if Case (2); 

b ' T ' CT ' |{0,...,^W}, if Case (3); 

J6,...,^W), if Case (4). 

Note that is omitted in the set X^ T<T . The main results of |JLZ12] 
can be summarized as follows. 

Theorem 5.1. Assume that no > 0. Let o be an irreducible generic 
cuspidal automorphic representation ofG no (A), andr be an irreducible 
unitary self-dual cuspidal automorphic representation of R_p'/_p(a)(A). 
Then the following hold. 

(1) The normalized Eisenstein series £^*(^a(t,6)®o-) s ) ^ s holomor- 
phic for Re(s) > except at s = so G X^ T a where it has possibly 
at most simple poles. 

(2) For So G (0, ^p], assume that the normalized Eisenstein series 
EaCi^Air^a, s) has a simple pole at s = s . Then the residue 
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o/£ , ^*(0 A ( Tj6 ) (g)(T , s) at s is square-integrable except s = in 
Case (3). 

(3) The global Arthur parameter for each of those residual repre- 
sentations is given in [JLZ12[ Section 6.2]. 

Note that the uq = case is given in |JLZ12l Theorem 5.2]. In 
|JLZ12] . the theorem was proved for the cases when G n is not uni- 
tary. However, it is expected that the same argument works for unitary 
groups as long as some technical results from |Arl2] are also valid for 
unitary groups ( |Mkl2j ). . 

The residual representations at the right end points of X^ Ta or of 
X^ T if no = are the candidates currently used to produce the auto- 
morphic kernel functions in the explicit constructions of endoscopy 
correspondences for classical groups. It is the point of [JL Z12] that 
the other residual representations may also be able to produce auto- 
morphic kernel functions for other kinds of Langlands functorial 
transfers. One example of this kind is referred to [G JSllj . 

5.2. Certain families of Fourier coefficients. Fourier coefficients 
of automorphic forms on classical groups considered here are attached 
to a particular family of partitions of type [d c l*]. The pair of d and c 
satisfies a parity condition according to the type of the classical group 
G m . More precisely one has the following cases. 

(1) When d — 2k — 1, c = 2f if G m is a symplectic group; and 
c = 2f or c = 2f + 1 if G m is an orthogonal group or unitary 
group. 

(2) When d = 2k, c = 2f if G m is an orthogonal group; and c = 2f 
or c = 2f + 1 if G m is a symplectic group or unitary group. 

As in | WO 1| Chapitre I] the set of the representatives of the adjoint 
G n (F)-orbits in the given stable orbit Cp cl *] associated to the partition 
[d c l*]can be explicitly given. 

Symplectic Group Case. Assume that G m is the symplectic group 
Sp 2m . If d — 2k — 1, then the integer c = 2f must be even, By 
| WO 1| Chapitre I], there is only one F-rational G m (F)-adjoint orbit in 
the given stable orbit 0^ cl2m _ cd ^ associated to the partition [d c l 2m ~ cd ], 

which corresponds to the pair ([d c l 2m ~ cd ], (•, •)). The notation q := (•, •) 
means that there is no quadratic forms involved in the parametrization 
of F-rational G m (F)-adjoint orbits in the given stable orbit Cy cl2m _ cd ,. 
To indicate the dependence on q, Vx q is used for Vx, and ipx q for ipx, 
respectively. The stabilizer in L of the character ip Xq is isomorphic to 

Sp c x Sp 2m 

— cd' 
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Hence, for any automorphic form ip on G m (A), the -^x, -Fourier coeffi- 
cient ^ Xq -(p) of (p, as defined in Section 4.2, is automorphic on 

Sp c (A) x Sp 

2m— cd 

(A). 

Remark 5.2. For a general automorphic form ip on G m (A), one does 
not expect that the ip Xq -Fourier coefficients F i(ip)(h, g) produces a 
reasonable (functorial) relation between automorphic forms on Sp c (A) 
and automorphic forms on Sp 2m _ cd (A) ; by simply assuming that the 
following integral 

[ [ ^ x i(p)(h,g)ip a (h)^)dhdg 

JS Pc (F)\S Pc (A) </Sp 2m _ cd (F)\Sp 2m _ cd (A) 

converges and is non-zero for some choice of data, where a G -4. cusp (Sp c ) 
and ir G Aus P (Sp 2m _ cd ). 

However, if one follows the idea of the classical theta correspondence 
using the minimality of the classical theta functions or the Weil repre- 
sentations, the ^Xq-Fourier coefficients J^ x s.((p)(h, g) do produce, via 
the integral above, reasonable functorial relations when one puts enough 
restrictions on the automorphic forms p. This will be discussed in more 
detail in Sections 5.3 and 5.4- 

ff d = 2k, then c = 2f or 2f + 1 is any positive integer. Fol- 
lowing | WO 1| Chapitre I], the set of the F-rational G m (F)-adjoint 
orbits in the stable orbit (9f* cl(2m _ cd) j are parameterized by the pairs 

([d c l (2m - cd) },(q d ,-)), where q d runs through the set of all F-equivalence 
classes of the c-dimensional non-degenerate quadratic forms over F. In 
this case, put q := (q d , •). The stabilizer in L of the character ip Xq is 
isomorphic to 

SO c (q d ) x Sp 2m 

— cd' 

Note that in this case, the space Q\ in (4.1) is nonzero, which leads the 
consideration of the Fourier-Jacobi coefficients in the construction 
of automorphic kernel functions, instead of using the t/>x,j- Fourier 
coefficients. This will be discussed in detail below. 

Even Special Orthogonal Group Case. Assume that G m is an 
F-quasisplit even special orthogonal group SO(gv) defined by the 2m- 
dimensional non-degenerate quadratic form qy. 

If d = 2k is an even positive integer, then c = 2f must be an even pos- 
itive integer. Following [W01, Chapitre I], there is only one F-rational 
G m (F)-adjoint orbit in the given stable orbit 0^ cl{2m _ cd) -^ associated to 

the partition [d c l^ 2m ~ cd ^], corresponding to the pair ([d c l^ 2m ~ cdS) ], (•, qi)), 
where q\ is the (2m — cd)- dimensional non-degenerate quadratic form 
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over F sharing the F-anisotropic kernel with qy. Put q := (•, q\). Then 
the stabilizer in Lq of the character ipx q is isomorphic to 

Sp c x S0 2m - C d(gi)- 

Note that in this case, the space Q\ in (4.1) is nonzero, which leads the 
consideration of the Fourier-Jacobi coefficients in the construction 
of automorphic kernel functions, instead of using the ?/>x,j- Fourier 
coefficients. This will be discussed in detail below. 

If d = 2k — 1 is an odd positive integer, then c = 2f or 2f + 1 is 
any positive integer. According to |W01t Chapitre I], the set of the 
F-rational G m (F)-adjoint orbits in the stable orbit Cf* cl(2m _ cd )i are pa- 
rameterized by the pairs ([d c l 2m ~ cd ], (qj, gi)), where q d runs through 
the set of all F-equivalence classes of the c-dimensional non-degenerate 
quadratic forms over F and qi runs through the set of all F-equivalence 
classes of the (2m — cd)- dimensional non-degenerate quadratic forms 
over F such that the quadratic forms qd®qi have the same F-anisotropic 
kernel as qy. Put q := (qd,qi)- The stabilizer in Lq of the character 
ipx q is isomorphic to 

SO c (g d ) x S0 2m - C d(gi). 
For any automorphic form ip on G m (A), the ipx q -Fourier coefficient 
ip, as defined in Section 4.2, is automorphic on 
SO c (q d ,A) x S0 2m _ cd (gi,A). 

Odd Special Orthogonal Group Case. Assume that G m is an 
F-split odd special orthogonal group S02 m +i((7v)- 

If d = 2k is an even positive integer, then c = 2f must be an even 
positive integer. Following |W01t Chapitre I], there is only one ir- 
rational G m (F)-adjoint orbit in the given stable orbit C?* cl(2m+1 _ cd) , 

associated to the partition [d c l^ 2m+1 ~ cd ^], corresponding to the pair 
Q £ ^c-j_(2m+i-cd)j ) ^ gjj^ where qi is the (2m + l — c<i)-dimensional F-split 
quadratic form over F. In this case, put q := (-,(?i). The stabilizer in 
L of the character ipx q is isomorphic to 

Sp c x S0 2m+ i- c d(gi). 

Note that in this case, the space Q\ in (4.1) is nonzero, which leads the 
consideration of the Fourier-Jacobi coefficients in the construction 
of automorphic kernel functions, instead of using the t/>x,,- Fourier 
coefficients. This will be discussed in detail below. 

If d = 2k — 1 is an odd positive integer, then c = 2f or 2f + 1 
is any positive integer. According to |W01l Chapitre I], the set of 
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the F-rational G m (F)-adjoint orbits in the stable orbit 0fj cl 2m+i-cdi 
are parameterized by the pairs ([d c l 2m+1 ~ cd ], (q d , qi)), where q d runs 
through the set of all F-equivalence classes of the c-dimensional non- 
degenerate quadratic forms over F and q\ runs through the set of all 
F-equivalence classes of the (2m + 1 — cd)- dimensional non-degenerate 
quadratic forms over F such that the quadratic forms q d © q\ are F- 
split. Put q := (q d ,qi)- The stabilizer in Lq of the character %px q is 
isomorphic to 

SO c (g d ) x S0 2m +i-cd(gi). 
For any automorphic form if on G m (A), the ipx q -Fourier coefficient 
F^(ip) of (p, as defined in Section 4.2, is automorphic on 
SO c (g d ,A) x S0 2m +i-cd(gi, A). 

Unitary Group Case. Assume that G m is a F-quasisplit unitary 
group U(my) = Uf;/F(mv) = U mv (gy) defined by an my-dimensional 
non-degenerate hermitian form qy with m = [^-] and assume that k 
is the sign of endoscopy data (U(my),£ x J of R,E/i?(my). By [WOlj 
Chapitre I], the set of the adjoint G m (F)-orbits in the stable orbit 
Of d oi(m v -cd)} are parameterized by the pairs ([d c l (jnv ~ cd ^], (q d , q{)). Here 
Q '■= {Qd, Qi) can be more specified as follows: 

(1) When d — 2k — 1, q d runs through the set of all F-equivalence 
classes of the c-dimensional non-degenerate hermitian forms 
over F and q% runs through the set of all F-equivalence classes 
of the {my — c<i)-dimensional non-degenerate hermitian forms 
over F such that the hermitian forms q d © q% have the same 
F- anisotropic kernel as qv; and 

(2) when d = 2k, q d runs through the set of all F-equivalence classes 
of the c-dimensional non-degenerate hermitian forms over F and 
qi is the (my — cd)- dimensional non-degenerate hermitian form 
over F such that q\ shares the same F-anisotropic kernel as qy. 

The stabilizer in L$ of the character ipx„ ls isomorphic to 

U c (g rf ) x U mv _ cd (gi). 

For any automorphic form ip on G m (A), the ipx q -Fourier coefficient 

F^ x i&) of (p, as defined in Section 4.2, is automorphic on 

U c (g d ,A) x U mv _crf(gi,A). 

Note that when d = 2k, the space Qi in (4.1) is nonzero, which leads the 
consideration of the Fourier-Jacobi coefficients in the construction 
of automorphic kernel functions, instead of using the ^x^-Fourier 
coefficients. This will be discussed in more detail below. 
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Note that the notion of Fourier^coefficients and related results hold 
for the metaplectic double cover Sp 2n (A) of Sp 2n (A). 

5.3. Automorphic kernel functions: Case (d=2k-l). From the 
constructions of the family of Fourier coefficients, this section starts 
the discussion on the possible constructions of automorphic kernel 
functions, which produce certain types of endoscopy correspon- 
dences via integral transforms as in Conjecture 11.21 

Recall that for any global Arthur parameter if), the automorphic L 2 - 
packet n^(e^) of the global Arthur packet U.^ consists of all elements 
Ti = <g)„7r„ G whose characters are equal to e^. The [<i c l*]-type 
Fourier coefficients are applied to the automorphic representations of 
G m (A) belonging to the automorphic L 2 -packet LT^ (e^ ) for a relatively 
simple global Arthur parameter if> , which will be specified below in 
each case. Of course, it is important to know that such an automorphic 
L 2 -packet IT^ (e^ ) is not empty, which essentially follows from the work 
of |JLZ12j and the discussion in Section 5.1. Certain refined structures 
on the cuspidal members in n^ (e^ ) have been studied first in [GJS12] 
and more recently in |Liul3j and |JL13j for some classical groups. It is 
expected that those work can be extended to all classical groups. The 
structures of Fourier coefficients (Conjectures 14.21 and 14. 3[) for such an 
automorphic L 2 -packet U.^ (e^ ) can be proved as discussed in Section 
4.4. 

Assume that r G ^4 C us P ( a ) is self-dual with a = 2e even, and assume 
that 

d = a — l = 2e — 1 

when consider the [c? c l*]-type Fourier coefficients. The other cases of 
the pairs (a, d) are considered in a forthcoming work of the author 

(M). 

Symplectic Group Case. Assume that G m is the symplectic group 

SP2m- 

Write 2m = a(b + c) with a = 2e > 2 being even and b > 0, c > 1 
being integers. For a self-dual r G ^4 C us P (a), if t is of symplectic type, 
take b + c to be even and hence b and c are in the same parity; and if 
t is of orthogonal type, take b + c to be odd and hence b and c are in 
the different parity. Consider a global Arthur parameter 

iP := (r,6 + c)ffl(l,l) 

of Sp 2m , which is the lift of the global Arthur parameter (r, b + c) of 
simple type for S02 m , via the natural embedding of the corresponding 
dual groups. 
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For the partition [d c l afe+c ] for Sp 2m with d = a — 1 odd, one must 
have that c = 2f is even. Hence if r is of symplectic type, then b = 21 
is even; and if r is of orthogonal type, then b = 21 + 1 is odd. It is 
not difficult to check that the automorphic L 2 -packet n^ (e^ ) of G m 
is not empty. In fact, it follows directly from Theorem 5.1 when r is of 
symplectic type and b + c is even. When r is of orthogonal type and 
b + c is odd, one may also construct a nonzero residual representation in 
the automorphic L 2 -packet n^ (e^ ). To do so, one needs to construct 
an irreducible cuspidal automorphic representation e of Sp 2e (A) from 
t which has the global Arthur parameter 

^ = (r,l)ffl(l,l). 

This is done by the composition of the automorphic descent from r of 
GL 2e (A) to SC>2e(A) with the theta correspondence from SC>2e(A) to 
Sp 2e (A). 

Take a member 6 in the automorphic L 2 -packet n^ (e^ ) of G m . The 
ipx q -Fourier coefficients J r ^ x ^(ifQ) of y?e € @, as discussed in Section 
5.2, are automorphic as functions restricted to the stabilizer 

S Pc (A) x S Pab+c (A). 

Define the automorphic kernel functions in this case to be 

(5.5) lC^(h,g):=^ x ^e)(h,g) 

for (h,g) G Sp c (A) x Sp ab+C (A). This family of the automorphic kernel 
functions JC^° e (h,g) should produce the endoscopy correspondence 

SO a6 (A) x S Pc (A) -> S Pa6+c (A) 

for Case (a=2e) of symplectic groups. Note that Sp ab+C is the sym- 
plectic group Sp 2n with 2n = ab + c and c = 2f as discussed in Section 
3.2; and SO a & is an F-quasisplit even special orthogonal group. Note 
that when 6 = 0, this is just the identity transfer from Sp c (A) to itself. 

More precisely, take ipi := (r, b). It is clear that ip\ is a simple global 
Arthur parameter of SO a j,. Now one can easily see that the global 
Arthur parameter vpo is built from the global Arthur parameter ipi of 
Go = SO a & and the structures of the groups H = Sp c and G = Sp a6+C . 
This construction of the automorphic kernel functions and their related 
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endoscopy correspondence can be described by the following diagram: 

SPa(b+c) 

G n^(e^) 

/ I 

GL a ,r >Ct e (h,g) 

/ I 
SO ab x Sp c < — > Sp a6+C 

n( T ,b)(e( T) 6)) n^, 2 (e^ 2 ) ^(T,b)^ 2 ( e (T,b)m^ 2 ) 

where a = 2e, c = 2f, and 2n = ah + c. When r is of symplectic type., 
b = 21 is even; and when r is of orthogonal type, h = 21 + 1 is odd. 

The main conjecture (Conjecture 11.21) specializes to the following 
conjecture for the current case. 

Conjecture 5.3 (Sp 2n : Case (a=2e)). For integers a = 2e > 2, 
b > 0, c = 2f > 2, and d = a — 1, let r G *4. CU sp(o.) be self-dual and 
ipi = (r, b) be a simple global Arthur parameter of an F-quasisplit SO a {,; 
and let ip2 and ip := ip± EH ip2 be global Arthur parameters of Sp c and 
Sp a6+c; respectively. For a G *4. 2 (Sp c ) and n G „4 2 (Sp afe+c ), if there 
exists an automorphic member G n^ (e^ ) ; such that the following 
integral 

I I JC%l(h,g)tp a (h)^Mi)dhdg 

JS Pc (F)\S Pc (A) JS Pai)+c (F)\Sp ai)+c (A) 

is nonzero for some choice of (p@ G 0, <p a G a, and ip n G it, assuming 
the convergence of the integral, then o G Il^ 2 (e^ 2 ) if and only if it G 

n^,(e^). 

Note that when r is of orthogonal type and 6=1 and when the global 
Arthur parameter ip 2 is generic, the construction of the endoscopy 
transfer for irreducible generic cuspidal automorphic representations 
was outlined in |G08] and see also |G12j . 

Example 1. A more explicit explanation of this construction is given 
for a phototype example. Take a standard parabolic subgroup P c e-i of 
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Sp a (fc+c) whose Levi decomposition is given by 

The unipotent radical V c e-i, which is the same as Vx q , have the struc- 
ture that 

V c e-i/[V c e-i, V c e-i] = Mat® (e ~ 2) © Mat c , n © Mat c © Mat c , n , 

where Mat c is the space of all c x c-matrices and Mat Ci „ is the space of 
all c x n-matrices with n = eb + f. The projection of elements v G V c e-i 
is given by 

(Xi,--- , X e _2, Yi, Y 2 , 13). 
For a nontrivial additive character ip F of F\A, the corresponding char- 
acter ip Xq on V c e-i(A) is defined by 

(5.6) := MHXi + • • • + X e _ 2 + Y 2 )). 

Then the stabilizer of the character t/j Xq in the Levi subgroup M c e-i is 

(5.7) S Pc x S Pafe+c H- (GL?- 1 ) x S Pa6+2c ) 
which is given by the embedding 

(/^^(/^(e-i) f h U 

\93 94/ 

where h 1— > h A ^ e ~^ is the diagonal embedding from Sp c into GL ( ? <6 ~ 1 ' ) , 

and9 =fe ?l) eSp — 

Even Special Orthogonal Group Case. Assume that G m is an 
F-quasisplit even special orthogonal group S0 2m (<2v) defined by the 
2m-dimensional non-degenerate quadratic form qy. 

Write 2m = a(b + c) with a = 2e > 2 being even and b, c > 1 being 
any positive integers. For a self-dual r G -4. C us P ( a )) take a simple global 
Arthur parameter 

ipo ■= (t, b + c) 

for S0 2m (<2v)- Hence if r is of symplectic type, then b + c is even and 
hence b and c are in the same parity; and if r is of orthogonal type, 
then b + c is odd and hence 6 and c are in the different parity. 

Assume that c = 2f + 1 is odd. If r is of symplectic type, then 
b = 21 + 1 is odd; and if r is of orthogonal type, then b = 21 is even. 
Consider the partition [d c l ( - ab+c ^] with d = a — I. Each F-rational 
adjoint G m (F)-orbit in the stable orbit 0** cl(ab+c) ^ corresponds to a 

pair ([d c l( ab+c ^], (q,i, qi)) as discussed in Section 5.2. Put q := (g<f,gi). 
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Following Theorem 5.1, the automorphic L 2 -packet n^, (e^ ) of G m is 
not empty. Take a member G in U^ Q (e^ ). The ipx q - Fourier coefficients 
(fe G 6, as discussed in Section 5.2, are automorphic as 

functions on 

SO c (g d )(A) x SO afc+c (g 1 )(A). 
Define the automorphic kernel functions in this case to be 

(5.8) K^(h,g):=F^ e )(h,g) 
for (h, g) G SO c (grf, A) x S0 O 6 +c (gi, A). This family of the automorphic 
kernel functions K.^o +c \h, g) should produce the endoscopy correspon- 
dence 

SO afe+ i(g ,A) x SO c (g d ,A) ->• SO ab+c (q u A) 

for Case (a=2e). Note that for the endoscopy group in £ sim (a6+c— 1), 
one takes SO a 6 +c (gi) to be an F-split odd special orthogonal group with 
2n + 1 = ab + c and c = 2f + 1, and hence both SO a f> + i and SO c are F- 
split odd special orthogonal groups as discussed in Section 3.1. Recall 
from Section 5.2 that the condition in general is that the quadratic 
form q d © q 1 shares the anisotropic kernel with the quadratic form q v 
defining S02 m . Hence the formulation here is more general than the 
elliptic endoscopy transfer for the F-quasisplit case, and is related to 
|Arl2| Chapter 9] for the transfers for inner forms. 

More precisely, take ipi := (r, b). It is clear that ip\ is a simple global 
Arthur parameter of SO a b + i(go) and the global Arthur parameter ip is 
built from the global Arthur parameter of Go = SO a & + i(go) and the 
structures of the groups H = SO c (gd) and G = SO a b +c (qi). The main 
conjecture (Conjecture II. 2p specializes to the following conjecture for 
the current case. It is clear that when b = 0, it produces the identity 
transfer from SO c to itself; and when c = 1 and 6 = 1, this construction 
reduces to the automorphic descent from GL a to SO a+ i as discussed in 
[GRSllJ. The more general case of c = 1 will be discussed in Section 
6. 

Conjecture 5.4 (S0 2n +i: Case (a=2e) and (c=2f+l)). For in- 
tegers a = 2e > 2, b > 0, c = 2f + 1 > 1, and d = a — 1, let 
t G A cusp (a) be self-dual and ipi = (r, 6) be a simple global Arthur 
parameter of F -split SO a b + i(go); and let ip 2 and ip :— ipi ffl ip2 be 
global Arthur parameters of SO c (qd) and SO a b+ c (qi) , respectively. For 
o~ G A2(SO c (qd)) and n G ^.2(SO a 6 +c (gi)), if there exists an automor- 
phic member G H^ {e^ ), such that the following integral 

I [ K^{h,g)^{h)^g)dhdg 

JSO c (q d ,F)\SO c (q d ,A) ^SO ai)+c (gi,F)\SO ai , +c (gi,A) 
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is nonzero for some choice of tps G 0, <p a G (7, and <p n G 7r ; assuming 
the convergence of the integral, then a G n^, 2 (e^, 2 ) i/ and only if n G 

n^(e^). 

This construction of the automorphic kernel functions and their re- 
lated endoscopy correspondence can be described by the following dia- 
gram: 

SO a ( 6+c )(qy) 

6 G Il^ (e^ ) 

/ I 
GL a ,r K*%g) 

/ I 
SO ab+ i(g ) x SO c (g d ) < — > SO afe+c (gi) 

n( T ,6)(e(r,b)) n^ 2 (e^ 2 ) ri( T)b ) H ^ 2 (e( T) 



where a = 2e, c = 2/ + 1, and 2n = afr + c — 1. When r is of symplectic 
type, b = 21 + 1 is odd; when r is of orthogonal type, b = 21 is even. 

Note that when r is of symplectic type and 6=1 and when the global 
Arthur parameter ip 2 is generic, the construction of the endoscopy 
transfer for irreducible generic cuspidal automorphic representations 
was outlined in [G08] and see also |G12j . Also when r is of symplectic 
type and b = c = 1, this can be viewed as a case of the automorphic 
descent from GL 2e to S02 e +i and was completely proved in jGRSllj . 
The general case of this conjecture is the subject discussed in [JZ13J. 

Assume now that c = 2f is even. If r is of symplectic type, then 
b = 21 is even; and if r is of orthogonal type, then b = 21 + 1 is odd. 
Consider the partition [<i c l( a6+c )] with d = a — 1. Each F-rational 
adjoint G m (F)-orbit in the stable orbit 0?^ cl(ab+c) -, corresponds to a 

pair ([d c l( ab+c ^], (qd,Qi)) as discussed in Section 5.2. Put q := (qd,qi), 
and for 6 G U.^ (e^ ) of G m , the i/^,,- Fourier coefficients J-*^ x -((fie) of 
(^e G 6, as discussed in Section 5.2, is automorphic on 

SO c (g d )(A) x S(W( gi )(A). 

Note that the automorphic L 2 -packet Il^ (e^ ) is not empty by Theo- 
rem 5.1. The automorphic kernel functions in this case are defined 
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to be 

(5.9) K&(h,g):=J* x ±{<pe){h,g) 

for (h, g) G SO c (grf, A) x S0 6 +C (gi, A). This family of the automorphic 
kernel functions K^q (h, g) should produce the endoscopy correspon- 
dence 

SO ab (g ,A) x SO c (g d ,A) -> SO ab+c (gi,A) 

for Case (a=2e). Note that for the endoscopy group in £ sim (a5 + c) 
with a = 2e and c = 2f, one takes SO a b+c(qi) to be an F-quasisplit 
even special orthogonal group, and hence both SO a ft(q ) and SO c (qd) 
are F-quasisplit even special orthogonal groups. Recall from Section 
5.2 that the condition in general is that the quadratic form qd@qi shares 
the anisotropic kernel with the quadratic form qv defining SC>2 m , which 
implies that rj qd ■ rj qi = rj qv , following the definition of character rj in 
[Arl2| Section 1.2]. Hence in order for SO a b(go) x SO c (gd) to be an 
elliptic endoscopy group of SO a & +c (gi), one must have 

Vqi = Vqo ' Vqd • 

Hence one must put the condition that 

Vqv = ^qd ' Vqi = Vqo ' Vq d = Vqo 

to make SO a &(go) x SO c (qd) an elliptic endoscopy group of SO a b +c (qi) 
as discussed in Section 3.3. Hence the formulation here is more general 
than the elliptic endoscopy transfer for the F-quasisplit case, and is 
related to |Arl2| Chapter 9] for the transfers for inner forms. 

More precisely, take ipi '■= { T , b)- It is clear that ip\ is a simple global 
Arthur parameter of SO a h(go) and the global Arthur parameter ip is 
built from the global Arthur parameter ipi of G = SO a b(g ) and the 
structures of the groups H = SO c (g ( i) and G = SO a b +c (qi). The main 
conjecture (Conjecture 11.21) specializes to the following conjecture for 
the current case. It is clear that when b = 0, it produces the iden- 
tity transfer from SO c (gd) to itself; and when c = 0, this construction 
reduces the identity transfer from SO a b(qo) to itself. The more gen- 
eral discussion of this endoscopy correspondence and its relation to the 
automorphic descents will be discussed in Section 6. 

Conjecture 5.5 (S02 n : Case (a=2e) and (c=2f)). For integers 
a = 2e > 2, b > 1, c = 2f > 2, and d = a — 1, let r G A cusp {a) be 
self-dual and ipi = (r, b) be a simple global Arthur parameter of F -split 
SO a f,(go); and let ip2 and if) := ipi EH ip2 be global Arthur parameters of 
SO c (gA and SO a b +c (qi) , respectively. For a G ^42(SO c (gA) and n G 
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-4.2(SO a b +c (gi)) ; if there exists an automorphic member G IL^ (e^ ) ) 
such that the following integral 

! I K%l{h,g) Va {h)i^g)dhdg 

JsO c (q dl F)\SO c (q d A) JS0 ab+c (qi,F)\S0 ab+c (qi,A) 

is nonzero for some choice of (p@ G 0, y CT G a, and ip n G tt, assuming 
the convergence of the integral, then o G IT^ 2 (e^ 2 ) if and only if 7r G 

n^,(e^). 

This construction of the automorphic kernel functions and their re- 
lated endoscopy correspondence can be described by the following dia- 
gram: 

SO a ( b+c )(qy) 
G Il^ (e^ ) 



GL Q ,r 



SO a6 (g ) x SO c (q d ) 
n( T , 6 )(e (T)6 )) n^, 2 (e^ 2 



« — ► SO a6+c (gi) 

n(T,fe)HV2 ( e (r,f>)ffli/>2 



where a = 2e, c = 2/, and 2n = ah + c. When r is of symplectic type, 
6 = 21 is even; when r is of orthogonal type, b = 21 + 1 is odd. 

This is the case has been discussed in |Jnllj . The details of the 
proof of this conjecture for the case when 6 = 2 and the global Arthur 
parameter ifj 2 is generic are given in the two papers in |GJS13j . When r 
is of orthogonal type and 6=1 and when the global Arthur parameter 
ip2 is generic, the construction of the endoscopy transfer for irreducible 
generic cuspidal automorphic representations was outlined in [G08J and 
see also |G12j . The general case of this conjecture will be the subject 
discussed in [JZ13j . 

Odd Special Orthogonal Group Case. Assume that G m is the F- 
split odd special orthogonal group S02- m +i- According to the discussion 
in Section 5.2 with d = 2k — 1, for any automorphic form ip on G m (A), 
the ipx q -Fourier coefficient J- s.{ip) of <p is automorphic on 

SO c (g d ,A) x S0 2m +i-cd(gi, A). 
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Note that the integers c and 2m+l— cd are in different parity. The func- 
torial meaning between automorphic forms A(SO c (qd)) and automor- 
phic forms A(SC>2m+i-cd(<li)) is not so easily related to the endoscopy 
structure in general and will be considered in the forthcoming work of 
the author QJnl3j). For Case (c=2f), it could be translated to the 
metaplectic double cover of Sp 2m (A) and formulated in the framework 
of endoscopy correspondence below. However, for Case (c=2f+l), 
it is more mysterious (see [Jnl3j ). except the case where c = 1 and 
6=1, which can be formulated as a case of automorphic descents of 
Ginzburg, Rallis and Soudry ( |GRSllj ) as follows. 

Assume that c = 1 and 6=1. Since a = 2e and d = 2e + 1 = 
a + 1, the partition is [(2e + l)l^ 2e ^] with m = a. This is the case 
that the automorphic descent takes r G ^4 cusp (2e) of orthogonal type 
to an cuspidal automorphic representation of S0 2e (A). More general 
discussion on automorphic descents will be reviewed in Section 6. The 
general situation of this case will be discussed discussion in |Jnl3j . 

Metaplectic Cover of Symplectic Group Case: Take G m (A) to 

be the metaplectic double cover Sp 2m (A) of Sp 2m (A). Consider that 
2m = a(b + c) with c = 2f > 2. As in the case of G m = Sp 2m , 
attached to the partition [d c l^ ab+c ^] with d = 2e — 1, the F-rational 
G m (F)-adjoint nilpotent orbit is F-stable. The associated ipx q - Fourier 

coefficient of any automorphic form ip on Sp 2m (A), J-^ x i((p), is auto- 
morphic on 

§5 C (A) x Sp a6+C (A). 

The meaning in endoscopy correspondence of this construction can be 
explained as follows. 

Let t G A cusp (a) be self dual. Assume that if r is of symplectic type, 
the central value £(§> 7r ) °^ ^ ne standard L-function (one could also use 
the partial L-function here) of 7r is nonzero. When r is of symplectic 
type, take 6 = 21 + 1; and when r is of orthogonal type, take 6 = 21. 
Then consider a global Arthur parameter 

ip := (r, 6 + c) 

Note that the dual group of Sp 2m (A) is considered to be Sp 2m and the 
theory of stable trace formula and the theory of endoscopy for Sp 2m (A) 
have been developing through a series of recent papers, including his 
PhD thesis of W.-W. Li ( |Liwllj ). although the complete theory is still 
in progress. Following the arguments in |JLZ12j . it is not hard to see 
that Theorem 5.1 holds for Sp 2m (A) and hence the automorphic L 2 - 
packet n^, (e^ ) is not empty ( |Liul3j ). For G Il^ (e^ ), Define the 



42 



DIHUA JIANG 



automorphic kernel functions in this case to be 



(5.10) 



for (h, g) G Sp c (A) x Sp a6+C (A). The endoscopy correspondence is given 



Note that this case can be regarded as a variant of the case S02 n +i 
with (a=2e) and 2n = ab + c as discussed in Section 3.3; and that the 
odd special orthogonal group SO a f,+i(go) is F-split. 

More precisely, take ipi '■= ( r ?^)- It is clear that ip\ is a simple 
global Arthur parameter of SO a b + i(g , A) or of Sp ab (A), and the global 
Arthur parameter if) is built from the global Arthur parameter ipi of 
Go = SO a b + i(go) or Sp ab (A), and the structures of the groups H = 
Sp c (A) and G = Sp afe+C (A). The main conjecture (Conjecture 11.21) can 
also be specialized to the following conjecture for the current case. 

Conjecture 5.6 (Sp 2n (A): Case (a=2e) and (c=2f)). Assume that 
integers a — 2e > 2, b > 1, c = 2f > 2, and d = a — I. Let r G *4 C usp( a ) 
be self-dual with the assumption that if t is of symplectic type, then 
L(|,7r) 7^ 0. Let ipi = (r,b) be a simple global Arthur parameter of 

F-split SO a b + i(g ? A) or of Sp ab (A); and let ip 2 an d 4> := 4>i ffl ip2 be 
global Arthur parameters of Sp c (A) and Sp a6+C (A) ; respectively. For 
cr G ^4.2(Sp c (A)) and Ti G *4.2(Sp afo+c (A)) ; if there exists an automorphic 
member G U.^ (e^ ), such that the following integral 



JS Pc (F)\S Pc (A) ^S Pab+c (F)\S Pab+c (A) 

is nonzero for some choice of ip® G 0, <p a G a, and ip n G n, assuming 
the convergence of the integral, then a G n^ 2 (e^, 2 ) if and only if n G 

n^,(e^). 



by 



SO ab+1 (g ,A) x Sp c (A) Sp afe+C (A), 



or by 



S Pa6 (A) x S Pc (A) S Pa6+c (A). 
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This construction of the automorphic kernel functions and their re- 
lated endoscopy correspondence can be described by the following dia- 
gram: 

SPa(b+c)( A ) 

G n^, (e^ ) 



GL a ,r 



SO afe+ i(g ,A) x Sp c (A) < — > 



where a = 2e, c = 2/, and 2n = ab + c. When r is of symplectic type, 
b = 21 + 1 is odd; when r is of orthogonal type, b = 21 is even. 
Some preliminary cases of this conjecture is considered in |JL13j . 

5.4. Automorphic kernel functions: Case (d=2k). The construc- 
tions of automorphic kernel functions with a = 2e + 1 > 1 being odd 
integers are considered here. In this case, the cuspidal automorphic 
representation r must be of orthogonal type. Following Waldspurger 
( |W014 Chapitre I], the structure of F-rational G m (F)-adjoint (nilpo- 
tent) orbits in the F-stable orbit 0?^,, associated to a partition of 
type [d c l*] was discussed in Section 5.2. Note that when d = 2k is 
even, the subspaces q±\ are non-zero. Hence there exists a generalized 
Jacobi subgroup, containing Stab^^x,) as the semi-simple part, of 
G m which stabilizes the character ip Xq - More precisely, following ( 14. ip . 
define V\ to be the unipotent subgroup of G m whose Lie algebra is 
©i>i0i. It is clear that Vi = Vx q is a normal subgroup of V\ and the 
quotient Vi/Vx q , which is abelian, is isomorphic to Let kerv 2 (ipx q ) 
be the kernel of the character ipx q in V 2 = Vx q - It is easy to check that 
Vi/ ker V2 (ip Xq ) is the Heisenberg group of dimension dimgx + 1. 

In order to define the Fourier- Jacobi coefficients of automorphic 
forms, let Qi := © be a polarization of the symplectic space 

Qi via the Lie structure of g and modulo ®i>2di- Let 9, - be the theta 
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function of the (generalized) Jacobi group 

(Stab^ x J x V 1 /ker V2 (^J)(A) 

attached to a Bruhat-Schwartz function <p G S(g,i(A)). For an auto- 
morphic form ip on G m (A), the ipx q -Fourier- Jacobi coefficient of ip is 
defined by 

(5.11) ?jl Xq -{ V )(x) := [ V (vx)el Xq -(vx)dv 

JVi(F)\Vi(A) 

where x G Stabi (ipx q ){A). 

Following |PS83] and [Ik94] , the basic theory of the Fourier- Jacobi co- 
efficients of automorphic forms asserts that for any irreducible automor- 
phic representation G A(G m ), the ipx q -Fourier coefficient, T ^(ye) 
is nonzero for some ip@ G O if and only if the following (generalized) 

Fourier- Jacobi coefficient TJ^ % {^>q){x) is nonzero for certain choice 
of data. It is this Fourier- Jacobi coefficient that helps the construction 
of the automorphic kernel functions for Case (d=2k). 

Symplectic group case. Assume that G m is the symplectic group 
Sp 2m . The partition is [d c l( 2m - cd )] with d = 2e. It follows that c = 
2/ > 2 or c = 2f + 1 > 1. Take 2m + 1 = a(b + c). Since r G A cusp (a) 
is of orthogonal type, b + c must be odd. Consider the global Arthur 
parameter 

ip := (r,b + c). 

By Theorem 5.1, the automorphic L 2 -packet n^, (e^ ) is not empty. 
Take a member 9 G n^ (e^, ) of G m (A). The Fourier- Jacobi coefficients 

? J ^ "(v 9 e)(^; 9) of <^e ^ @ define automorphic functions on 

SO c (g d ,A) x Sp~ 2n (A) 

with 2n = ab + c— 1. Note that this pair forms a reductive dual pair in 
the sense of R. Howe in the theory of theta correspondence and hence 
it is known that if c = 2f, the covering splits and one has 

S0 2/ (g d ,A) x Sp 2n (A); 
and if c = 2f + 1, the covering splits over SO c (q d , A) and one has 

S0 2 /+i(g d ,A) x Sp 2n (A). 

Note that by Section 3.2, the special orthogonal group SO c (qd) may 
not be F-quasisplit. 
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Define the automorphic kernel functions in this case to be 

(5.12) /Cj°(M):=^%e)(M) 

for (h, g) G SO c (gd, A) x Sp 2n (A), accordingly. This family of the au- 
tomorphic kernel functions K$ (h,g) should produce the endoscopy 
correspondence 

G (A)xSO c (g d ,A)^Sp' 2n (A). 
When c = 2f, which implies that b = 21 + 1, one has 

(5.13) Sp^A) x S0 2/ ( to A) Sp 2n (A). 

Take ip\ := (r, b) with b = 2/ + 1, which is a global Arthur parameter for 
Go = Spo^x. This construction of the automorphic kernel functions 
and their related endoscopy correspondence can be described by the 
following diagram: 

SPa(£>+c)-l 

G n^, (e^ ) 



GL a ,r 



Ktl{h,g) 



n( r ,6) 



X SO c (q d ) 

n^2 \^ip2, 



i — y 



SPafe+c-l 



where a = 2e + 1, c = 2/, b — 21 + 1, 2n = ab + c — 1, and r is of 
orthogonal type. The main conjecture (Conjecture 11.21) specializes to 
the following conjecture for the current case. Note that if c = 2f = 0, 
it is the identity transfer for Sp ab _ 1 (A). 

Conjecture 5.7 (Sp 2n : Case (a=2e+l) and (c=2f)). Assume that 
integers a = 2e + 1 > 1 with d = a — 1, and b = 21 + 1 > 1 and 
c = 2f > 0. Le£ r G *4 C usp(a) o/ orthogonal type. Let ip\ = (r, b) be 
a simple global Arthur parameter of for Sp ob _ 1; ip2 be a global Arthur 
parameters of SO c (qd), andip := ipiSifj 2 be the global Arthur parameter 
for G n (A) := Sp 2n (A) ; respectively. For o G A^(^0 ^qa)) and it G 
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>A.2{G n ), if there exists an automorphic member 6 £ U^ (e^ ), such 
that the following integral 



K^ Q {h,g)ip<r{h)^{g)dhdg 

SO c (q d ,F)\SO c {q d ,A) J G n (F)\G n (A) 

is nonzero for some choice of (p@ £ 0, ip a £ a, and ip w £ it, assuming 
the convergence of the integral, then o £ n^ 2 (e^ 2 ) if and only if n £ 

n^(e^). 

Note that when 6=1 and the global Arthur parameter tp2 is generic, 
an outline of discussion of this conjecture for endoscopy transfer of 
irreducible generic cuspidal automorphic representations was given in 
[G08] and see also [Gl2] . 

When c = 2f + 1, which implies that b = 21, one has 

(5.14) SO ab+1 (q , A) x S0 2/+1 (g d , A) ->■ Sp 2n (A). 

This is exactly the endoscopy transfer introduced by Wen-Wei Li in 
|Liwll] when both SO a b+i(qo) and SC^j+i^d) are F-split. Recall from 
Section 3.2 that in general SO a & + i(go) and S02/+i(<?d) may not be F- 
quasisplit. This general case will be discussed elsewhere. 

Take ipi := (r, b), which is a global Arthur parameter for SO a ;, + i(go)- 
The construction of the automorphic kernel functions and their related 
endoscopy correspondence in this case can be described by the following 
diagram: 

SPa(b+c)-l 

6 £ n^ (e^, ) 

/* I 
GL a ,r £J°(M) 

/ I 

SO ai ,+i(g ) x SO c (g d ) < — > Sp^+^A) 



where a = 2e + 1, c = 2f + 1, 6 = 2/, 2n = ah + c — 1, and r is of 
orthogonal type. 
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The main conjecture (Conjecture 11.21) specializes to the following 
conjecture for the current case of endoscopy correspondences. 

Conjecture 5.8 (Sp 2n : Case (a=2e+l) and (c=2f+l)). Assume 
that integers a = 2e + 1 > 1 with d = a — 1, and b = 21 > 2 and 
c = 2f + 1 > \, and assume that r G A cusp (a) is of orthogonal type. 
Let ipi = (r, b) be a simple global Arthur parameter for SO a fe + i(g ); 
ip2 be a global Arthur parameter of SO c (qd), and ip :— tpx ffl ip 2 be the 
global Arthur parameter for G n (A) := Sp 2n (A), respectively. For a G 
-4.2(SO c (gd)) and tt G A2{G n ), if there exists an automorphic member 
G U^ (e^ ), such that the following integral 



Ig, 



^ e (h,g)<p a (h)ipMdhdg 

SO c (q d ,F)\SO c (q d ,A) J G n (F)\G n (A) 

is nonzero for some choice of ip@ G @ ; if a G o, and ip n G it, assuming 
the convergence of the integral, then o G Il^ 2 (e^ 2 ) if and only if ft E 

n^,(e^). 

Some preliminary cases of this conjecture will be treated in |JL13j . 

Example 2. A more explicit explanation of the construction of the 
Fourier- Jacobi coefficients is given for a phototype example. Let r G 
A CVLSp (a) with a = 2e + 1 be self-dual. Then r must be of orthogonal 
type. Also b = 21 + 1 is odd and c = 2e is even. Assume that 2n = 
c + ab — 1. 

Consider a standard parabolic subgroup P c e of Sp fl ( c+fe - ) _ 1 , whose Levi 
decomposition is 

Q c e = M c eV c e = (GL* ^ X Sp^^)^ 

with 2n = ab + c — 1. The elements in the unipotent radical V c e, which 
is equal to Vi, are of form 



(5.15) 



I c X e _i * * 

hn Y* 
Ir 



V 



The subgroup consisting of all elements with Y = is a normal sub- 
group of V c e, which is equal to Vx q - The character ipx q of Vx q is given 
by 

(5.16) tp Xq (v) := ^(tr(Xx + • ■ ■ + X e _i) + tr(S q X )), 
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where S q is a non-degenerate c x c-symmetric matrix defining q. The 

stabilizer of the character if) Xq in the Levi subgroup GL*^ x Sp 2n is 
isomorphic to 

SO c (S £ ) x Sp 2n . 

The elements (h,g) of SO c (S q ) x Sp 2n is embedded into GL*^ x Sp 2n 
by 

(h,g)^(h^\g) 

where h h-> h A ^ is the diagonal embedding SO c {S q ) into GL** 6 -*. 

Even special orthogonal group case. Assume that G m is an 
F-quasisplit even special orthogonal group S0 2m (qy) defined by the 
2m-dimensional non-degenerate quadratic form qy. 

For the partition [d c l^ 2m ~ cd ^] for S0 2m (qy)> d = a — 1 = 2e is even 
and hence one must have that c = 2f is even. In this case, take that 
2m = a(b + c) + 1 with 6 + c odd and hence b = 21 + 1 must be odd. For 
t G ^4 CU sp(a) to be of orthogonal type, take the global Arthur parameter 

Vo:=M + c)ffl(x,l) 

for S0 2m (qy), which is the lift of the simple global Arthur parameter 
for Sp a ( b+C - ) _ 1 to SO a (6 +c ) + i(qy) via the theta correspondence, based on 
the theta lift from Sp a _ 1 to SO a+ i(g) with q and qy share the same 
anisotropic kernel. 

Following |JLZ12j . one can produce a nonzero residual representation 
in the automorphic L 2 -packet IT^ (e^ ), with cuspidal support 

(GL a x ^xSO a+1 (g),r^)® e ) 

where e is the image of r under the composition of the automorphic 
descent of GL a to Sp a _ 1 with the theta correspondence from Sp a _ 1 to 
SO a+ i. Take a member 6 in the set Il^ (e^ ) of G m . The Fourier- Jacobi 

coefficients TJ^ 1 ((fe)(h, g) of <^e £ @ define automorphic functions 
on 

Sp c (A) x SO o6+c+ i(<2i, A). 
Define the automorphic kernel functions in this case to be 

(5-17) /Cj°(M):=^f%e)(M) 

for (h, g) G Sp c (A) x SO a 6 +c+ i(gi, A). This family of the automorphic 
kernel functions lC^° e (h,g) should produce the endoscopy correspon- 
dence 

SPot-i(A) x S Pc (A) SO ab+c+ i(gi,A). 
Note that SO a b +c+ i(gi) is the F-quasisplit even special orthogonal group 
defined by qi, which shares the same F-anisotropic kernel with qy. 
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The main conjecture (Conjecture 11.21) specializes to the following 
conjecture for the current case. 

Conjecture 5.9 (S0 2n : Case (a=2e+l) and (c=2f)). Assume that 
integers a = 2e + 1 > 1 with d = a — 1, and b = 21 + 1 > 1 and 
c = 2f > 2. Let t G ^4 C us P (o) be of orthogonal type. Let ipi = (r,b) 
be a simple global Arthur parameter of for Sp ab _ 1 . Let %p 2 be a global 
Arthur parameters of Sp c and let ip :— ipi ffl ip2 be the global Arthur 
parameter for SO-2 n (A) with 2n = ab + c + 1. For a G ^(SpJ and 
n G *4.2(S02n); if there exists an automorphic member G U^ (e^ ), 
such that the following integral 

I I ^l(h,g)^(h)^)dhdg 

JSp c (F)\Sp c (A) JS0 2 n(F)\S02„(A) 

is nonzero for some choice of (p@ G 0, (p a G a, and <f^ G it, assuming 
the convergence of the integral, then a G LT^ 2 (e^ 2 ) if and only if it G 

n^(e^). 

This construction of the automorphic kernel functions and their re- 
lated endoscopy correspondence can be described by the following dia- 
gram: 

SO a ( fe+c ) + i(qy) 
G n^ (e^, ) 



GL a , r 



K&%9) 



SPab-1 



X 



Sp c 



n( T ,6)(e( rj6 )] 



n^ 2 (e^ 2 , 



< — > 



SO a fc+c+l(<7l, - 



n( T ,ft)EV>2 ( e (T,b)Hi/. 2 



where a = 2e + 1, c = 2/, b — 21 + 1, 2n — ab + c + 1, and r is of 
orthogonal type. 

Note that when 6=1 and the global Arthur parameter ip2 is generic, 
an outline of discussion of this conjecture was given for the endoscopy 
transfer of irreducible generic cuspidal automorphic representations in 
[G08J and see also |G12j . The general case is the subject that will be 
discussed in [JZ13j . 
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Odd special orthogonal group case. Assume that G m is F-split 
odd special orthogonal group S0 2m+ i. The partition is [dn^ m+1 - cd >] 
with d = a — 1 = 2e, and hence c = 2f. Take 2m = a(b + c) with 
b = 21. Consider the simple global Arthur parameter for S0 2m +i: 

%po := (r, b + c) 

with r G A CU s P (cl) being of orthogonal type. Take 9 G U^ (e^ ) of 

G m . The Fourier- Jacobi coefficients J- 'J ' ^ 1 ((p e )(h, g) of y?e £ ® define 
automorphic functions on 

Sp c (A) x SOot+c+itei, A). 
Define the automorphic kernel functions in this case to be 

(5.18) /Cj°(M):=^^%e)(M) 

for (h,g) G Sp c (A) x SO a 6 +c+ i(g , i, A). This family of the automorphic 
kernel functions fC^(h,g) should produce the endoscopy correspon- 
dence 

Sp^A) x Sp^(A) SO ab+c+1 { gi ,A). 

Note that SO a ;, +c+ i(gi) is the F-split odd special orthogonal group 
defined by q±. It is possible to consider this transfer for more general 
qv and q±. The main conjecture (Conjecture 11.21) specializes to the 
following conjecture for the current case of endoscopy correspondences. 

Conjecture 5.10 (S0 2n+ i: Case (a=2e+l) and (c=2f)). Assume 
that integers a = 2e + 1 > 1 with d = a — 1, and b = 2l,c = 2f>2. Let 
t G *4. C usp( a ) be of orthogonal type. Let ipi = (r,b) be a simple global 
Arthur parameter of for Sp afe (A). Let ip2 be a global Arthur parameters 
of Sp c (A) and let ip :— ip\ ffl ip2 be the global Arthur parameter for 
S0 2n +i(A) with2n = ab + c. For a G „4 2 (Sp c (A)) and it G .A 2 (S0 2n+ i), 
if there exists an automorphic member B G U^ (e^ ), such that the 
following integral 

[ _ / K^ e {h,g)^{h)^Mdhdg 

JS Pc (F)\S Pc (A) JS0 2 „+i (F)\S0 2 „+i (A) 

is nonzero for some choice of (p@ G 0, tp a G a, and G it, assuming 
the convergence of the integral, then a G Il^ 2 (e^ 2 ) if and only if n G 

n^(e^). 

The construction of the automorphic kernel functions and their re- 
lated endoscopy correspondence in this case can be described by the 
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SO a ( fe+c ) + i(gvO 

G 5^(6^) 

Sp a6 (A) x Sp c (A) < — ► SO a6+c+ i(gi,A) 

n( r ,6)(e(r,b)) n^ 2 (e^, 2 ) R{T,b)mip 2 (. e (T,b)mip 2 ) 

where a = 2e + 1, c = 2f, b = 21, 2n = ab + c, and r is of orthogonal 
type. 

Note that the discussion here assumes that d = a — 1, and see [Jnl3j 
for other possible choices of the pairs (a, d). 

5.5. Unitary group case. Assume that G m is an F-quasisplit unitary 
group U^/jr(mv) = U(my) = U mv (gy) defined by the my-dimensional 
non-degenerate hermitian form qy, with m = pr 1 ], and assume that /t 
is the sign of in the endoscopy data 

(UH,U = (U(mv),x«) 

of Rb/f(w)' Further, assume that d = a — 1 and that my = a(b+c) or 
my = a{b + c) + 1. Recall from Section 5.2 that the Fourier coefficients 
associated to the partition [rf c l( m v _cc 0] produce the stabilizer 

U c (g d ) x U mv _cd(gi). 

Assume in the rest of this section that U c (q | A x U mv ,_ c d(gi) is F- 
quasisplit and hence is denoted by 

U(c) x U(m,y — cd). 

The more general cases will be considered elsewhere. 

The (standard or twisted) elliptic endoscopy is to consider the fol- 
lowing transfer 

(5.19) U(my - ac) x U(c) -)■ U(my - dc). 

The signs K mv -ac, n c and n mv _ dc of the L-embedding for U(my — ac), 
U(c) and U(my — dc), respectively, can be determined as follows. It 
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is clear that U(my — dc) is part of the Levi subgroup of U(my), and 
hence 

^my—dc ^my • 

By |Mkl2l Section 2.4], one must have that 

K mv „ ac {-l) mv - aC - 1 = Kc(-l) C_1 = ^(-l)^^ 1 - 

It follows that 



my—ac 



(5.20) k c = k(-1] 

Therefore, the elliptic endoscopy considered here is to consider the 
transfer 

(5.21) {U{m v -ac)xXJ{c),{K{-l) c ,K{-l) mv - ac )) -)• (U(m v -dc), «). 

The automorphic kernel functions are constructed via automor- 
phic representations 9 belonging to the automorphic L 2 -packet II^ (e^ ) 
associated to the global Arthur parameter 



\(r,b + c) if m v = a(b + c); 

; ° ~ \(r,b + c) ffl (x,l) ifm v = a(6 + c) + l, 

where r is a conjugate self-dual member in A cnsp (a) and x is either 
the trivial character or ue/f the quadratic character attached to the 
quadratic field extension E/F by the global classfield theory, depending 
the parity of the data. 

The construction and the theory in Sections 5.3 and 5.4 work for 
unitary groups. The details will be presented in a forthcoming work of 
Lei Zhang and the author in |JZ13j and are omitted here. 

It is worthwhile to mention that the constructions outlined here may 
work for groups which are not necessarily F-quasisplit, which leads the 
endoscopy correspondences for inner forms of classical groups ( [Arl2| 
Chapter 9]). The constructions discussed here essentially assume that 
d = a — 1 and the remaining possible cases are discussed in the forth- 
coming work of the author ( |Jnl3j ). 

6. Automorphic Descents and Automorphic Forms of 

Simple Type 

As discussed in previous sections, the main idea to construct ex- 
plicit endoscopy correspondences for classical groups is to produce au- 
tomorphic kernel functions fct^(h,g) on H(A) x G(A) by means 
of automorphic forms of simple or simpler type on an ample group 
G m (A). The integral transforms with the constructed automorphic 
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kernel functions lCl®(h, g) produce the endoscopy correspondences: the 
endoscopy transfers from if (A) to G(A) and the endoscopy de- 
scents from G(A) to if (A). Hence a refined explicit construction the- 
ory of endoscopy correspondences requires refined knowledge about the 
automorphic L 2 -packets of simple or simpler type. The automorphic 
descent method has been developed by Ginzburg, Rallis and Soudry in 
1999 and completed in their book in 2011 ( |GRSllj ) for F-quasisplit 
classical groups, and its extension to the spinor groups is given by J. 
Hundley and E. Sayag in [HS12j . The generalization of the automor- 
phic descent method by Ginzburg, Soudry and the author ( |GJS12j ). 
and through the work of Baiying Liu and the author ( [JL13] ) and the 
work of Lei Zhang and the author QJZ13| ) provides an effective way to 
establish refined structures of automorphic L 2 -packets of simple type. 
The connections with the topics in the classical theory of automorphic 
forms: the Saito-Kurokawa conjecture and its relations to the Kohnan 
space and Maass space of classical modular forms, with the Ikeda lift- 
ings and the Andrianov conjecture in classical theory of automorphic 
forms will also be briefly discussed and more work in this aspect will 
be addressed in a future work. 

6.1. Automorphic descents. The method of automorphic descents 
and its connection with the Langlands functoriality between automor- 
phic representations of classical groups and general linear groups was 
discovered by Ginzburg, Rallis and Soudry in 1999, through their se- 
rious of papers published since then. Their book QGRSllJ) gives a 
complete account of their method for all F-quasisplit classical groups. 
The idea to construct such automorphic descents may go back to the 
classical examples in earlier work of H. Maass ( |Ms79j ). of A. Andrianov 
([An79] L of I. Piatetski-Shapiro ( |FS83j ). of M. Eic hler and D. Zagier 
[EZ85] ) . and of N. Skoruppa and D. Zagier ([SjZ88]), all of which are 
related to the understanding of the classical example of non-tempered 
cuspidal automorphic forms of Sp(4) by H. Saito and N. Kurokawa 
in 1977. The method in |GRSllj extends to the great generality the 
argument of Piatetski-Shapiro in [PS83J. 

In terms of the general formulation of the constructions of the auto- 
morphic kernel functions in Section 5, the automorphic descents of 
Ginzburg, Rallis and Soudry ( |GRSllj ) is to deal with the special cases 
corresponding to the partitions of type [d 1 !*] with c = 1 and 6=1. 
The following discussion will be focused on two cases of automorphic 
descents and concerned with their extensions and refined properties as 
developed in the work of Ginzburg, Soudry and the author ( |GJS12j ). 
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the PhD. thesis of Liu ( |Liul3j ) and a more recent work of Liu and the 
author ( [JLT5] ). 

Recall from Section 5.4 the formulation of the Fourier- Jacobi coef- 
ficients of automorphic forms in the special case corresponding to the 
partition of type [(2fc) l< 2m ~ 2fc )] withG m (A) = Sp 2m (A) or Sp 2m (A), and 
c = 1 and d = 2k. The Fourier- Jacobi coefficient defines a mapping 

(2m) 

FJ(2m-2k) from automorphic forms on Sp 2m (A) to automorphic forms 

on Sp 2m _ 2fe (A) an d a mapping ^ jf^m-2k) fr° m automorphic forms on 

Sp 2m (A) to automorphic forms on Sp 2m _ 2fc (A). In general, one does 
not expect that those mappings carry any functorial meaning for auto- 
morphic forms in the sense of Langlands. However, they do yield the 
functorial transfers when they are restricted to the automorphic forms 
of simple type in the sense of Arthur ( |Arl2j ). 

The discussion will be focused on one case when r 6 ^4 cusp (2e) has 
the property that it is of symplectic type and r) ^ for the stan- 
dard L-function L(s,r). Following |GRS11] . the automorphic descent 
construction can be reformulated in terms of the Arthur classification 
theory of discrete spectrum and described in the following diagram: 

S T G n( T) 2)H(i,l)(e( T)2 )EB(l,l)) 



7r G II( Ti i)(e( rj i)) 
This diagram of constructions can be reformulated as 

Sp 4e (r,2)ffl(l,l) 

(r,l) Sp 2e t 
\ 

Sp (1,1) 

which can be extended to the general version of the constructions: 

Sp 4e/+4e (r,2/ + 2)ffl(l,l) 

(6.2) (r,2/ + l) Sp 4e , +2e t 

\ 

Sp 4e , (r,2/)ffl(l,l) 



Sp 4e (A) 

RES y\ 

(6.1) r GL 2e (A) |FJ 

LFT \ 

Sp 2e (A) 
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This diagram is called a basic triangle of constructions. The dual 
triangle is the following: 

(r, 21 + 1) Sp 4e/+2e 

\ 

(6.3) t Sp 4ei (r,2Z)ffl(l,l) 

(r,2/-l) Sp 4e ;_ 2 ; 

Putting the two basic triangles of constructions, one obtains the fol- 
lowing general diagram with r symplectic, L(|,r) ^ 0: 

: t 
(r,2/-l) Sp 4ei _ 2e 
t 

(6.4) : j 

t 

(r, 3) Sp 6e 
t 

(t, 1) Sp 2e 

The matching of the global Arthur parameters in this diagram under 
the Fourier- Jacobi mappings has been verified in |JL13j . 

It is proved in |GJS12] that those two basic triangles are commuting 
diagrams for distinguished members in the packets. It is clear that 
for distinguished members of the packets, the vertical arrow-up is con- 
structed by taking residue of the Eisenstein series with the given cusp- 
idal datum. However, if one considers general members of the packets, 
the vertical arrow-up can be constructed by taking a special case of the 
endoscopy transfer considered in Section 5. Of course, the existence of 
such transfers is confirmed in |Arl2] (and also |Mkl2j ) as one step of 
the induction argument of Arthur. The commutativity of those basic 
triangles in general will be discussed in |JL13] . 



\ 
\ 



Sp 4ei (r,2/)ffl(l,l) 
t 

Sp 4ei _ 4e (r,2/-2)ffl(l,l) 



S P8e 
t 

Sp 4e 



(r,4)ffl(l,l) 



(r,2)ffl(l,l) 
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6.2. Finer structures of automorphic L 2 -packets of simple type. 

Diagram f)6.4p and its properties can be used to establish finer struc- 
tures of automorphic L 2 -packets of simple type in terms of of the struc- 
ture of Fourier coefficients in the spirit of Conj ectures 14 . 21 and 14 . 31 This 
is tested in |Liul3] and |JL13j . 

The following basic triangle was first considered in [GJS12] : 

(r, 3) Sp 6e 

\ 

(6.5) t Sp 4e (r,2) 

(t, 1) Sp 2e 

The motivation was to understand more precise structures of the trans- 
fers between Sp 2e (A) and Sp 4e (A). From the diagram, the transfer $ 

from Sp 4e (A) to Sp 2e (A) is given by the corresponding Fourier- Jacobi 
coefficient. On the other hand, the composition of the transfer from 
Sp 2e (A) to Sp 6e (A) with the transfer from Sp 6e (A) to Sp 4e (A) given 
by the corresponding Fourier- Jacobi coefficient yields a transfer \1/ di- 
rectly from Sp 2e (A) to Sp 4e (A). The question was to characterize the 
image under the transfer \1/ from Sp 2e (A) to Sp 4e (A) of the automorphic 

L 2 -packet IT( T| i)(e( T) i)) in au tomorphic L 2 -packet n( r ,2)ffl(i,i)(e( r ,2)ffl(i,i))- 
The main result of |GJS12] can be reformulated in terms of the Arthur 
classification as follows. 

Recall that the Fourier- Jacobi coefficients defined in Section 5.4 de- 
pends on an F-rational datum q. In the current case here, this q reduces 
to the square classes of the number field F. For simplicity in notation 
here, the square classes are denoted by a. This a defines a nontrivial 
additive character of F, denoted by ip a , i.e. ip a (x) := ipp(ax). To be 
more precise, the global Arthur packet 11^ for Sp 2e (A) depends on a 
choice of additive character of F. It is reasonable to denote by n^(e^) 

the automorphic L 2 -packet within the global Arthur packet IT^, whose 
members are ?/> Q -functorial transferred to the Arthur representation tt^ 
of GL 2e (A) as discussed in Section 2.2. When if) is a generic global 
Arthur parameter, ?/; Q -functorial transfer from Sp 2e (A) to GL 2e (A) was 
established in |GRS11] for the generic members in n^(e^). Of course, 
the whole theory on the Arthur classification of the discrete spectrum 
for Sp 2e (A) is not known yet, and is expected to be achieved through se- 
ries of publications of Wen- Wei Li started from his PhD. thesis |Liwllj . 
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Define nL^gn 1 )(e(r,2)ffl(i,i)) to be the subset of the automorphic 
L 2 -packet ri( rj 2)ffl(i,i)(e(r,2)H(i,i)) consisting of all members whose ip- a - 
Fourier-Jacobi descent as discussed above from n( T ,2)Ba(i,i)( e (T,2)ffl(i,i)) to 
U? ^(e( r ,i)) are non-zero. Hence one has the following disjoint union 
decomposition 
(6.6) 

n( T , 2 )ffl(l,l)(e( T ,2)ffi(l,l)) = n (r,2) ) ffl(l,l)( e (^ 2 ) ffl ( 1 . 1 )) U n (r,2)EB(l,l)( e (r,2)ffl(l,l)) 

where nt^^n i)( e (r,2)ffl(i,i)) i s defined to be the complimentary sub- 
set of the subset nLgpn y(e(r,2)ffl(i,i)) in the automorphic L 2 -packet 

n( T ,2)E(i,i)( e (T,2)ffl(i,i))- The main result of |GJS12j can be reformulated 
as follows in terms of the Arthur classification theory. 



Theorem 6.1 ( |GJS12j ). With notation as above, the transfers <3? a and 
ty a establish one-to-one correspondence between the setH? Tl Je^ T) x^) and 

the set n[~2)ii,i)(e(r,2)ffl(i,i))- 



In order to prove this theorem, one has to combine the basic triangle 
( 16. 5 1) with the basic triangle ( 16. 2 I) with 1 = 1. This leads to the following 
diagram of transfers: 

Sp 8e (r,4)ffl(l,l) 

(r,3) Sp 6e t 
(6.7) \ 

t Sp 4e (r,2)ffl(l,l) 

The commutativity of the transfers in this diagram plays essential role 
in the proof of Theorem 16.11 

It is worthwhile to make some remarks on the relation of this theorem 
to previous known works in automorphic representation theory and in 
classical theory of automorphic forms. First of all, this theorem can 
be regarded as a refinement of the main results of [GRS05], where 
the authors tried to construct, with a condition of the structure of 
certain Fourier coefficients, some cuspidal automorphic representations 
of Sp 2n (A) or Sp 2n (A) that are expected to belong to one of the global 
Arthur packets of simple type as discussed above. Those conditions 
will be discussed below related to a work of Liu and the author |JL13] . 
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Secondly, when e = 1 , this theorem is a refinement of the main result 
of Piatetski-Shapiro's pioneer paper QPS83J) on the representation- 
theoretic approach to the Saito-Kurokawa examples of non-tempered 
cuspidal automorphic forms on Sp 4 ; and can also be viewed as the 
representation-theoretic version of a theorem of Skoruppa and Zagier 
QSZ88J), which completed the earlier work of H. Maass ( |Ms79j ) and of 
A. Andrianov ( |An79] ) . For a complete account of this classical topic, 
the readers are referred to the book QEZ85J) and the survey paper 

(MR)- 

Finally, it will be an interesting problem to work out the explicit 
version in the classical theory of automorphic forms of this theorem, 
which will be the natural extension of the Skoruppa and Zagier theorem 
QSZ88J) to classical automorphic forms of higher genus. 

One of the motivations of the work of Liu and the author ( [Liul3] and 
[JL13] ) is to extend Theorem 16 .ll to the cases with general global Arthur 
parameters of simple type for Sp 4ei (A) and Sp 4eZ+2e (A), respectively. 
More precisely, one defines similarly the following decomposition 

(6.8) n4g = n(-» )+ (^)un«- n) -(g 

for Sp 4ei (A) with ip = (r, 21) EH (1,1), where Il^~ a ' 1+ (e^) consists of 

elements of IL^e^) with a non-zero ^_ a -Fourier-Jacobi descent from 

Sp 4ei (A) to Sp 4eZ _ 2e (A), and Ili' a '~(e^) is the complement of Irl~ a ' + (ef) 

in Il^(e^). 

Similarly, one defines 

(6.9) II( T> a + i)(e( T) a+i)) = nf+ 2l+1) (e( T , a+ i)) U n£~ 2 , +1) (e( Tj2 j+i)) 

for Sp 4ei+2e (A), where the subset n" T 4 2 i+i)( e (' i ": 2i + 1 )) consists of elements 
in II( T) 2j+i)(e(T,2j+i)) with a non-zero ^-Fourier- Jacobi descent from 
Sp 4d+2e (A) to Sp 4ei (A), and the subset U-?~ 2 i + x\(^(t,21+i)) is defined to 
be its complement. 

It is clear that further understanding of the structures of those sets 
and their mutual relations are the key steps to understand the struc- 
tures of Fourier coefficients of the automorphic representations within 

n( Tl2 Qffl(i,i)(e( T ,2i)ffl(i,i)) and n( T ,»+i)(e(T,»+i))> respectively. In |Liul3j, 
the cuspidal part of those sets has been more carefully studied, which 
is an extension of the main result of [GJS12j . 

The problems have been formulated for other classical groups ac- 
cording to the functorial type of r. The details of the progress in this 
aspect will be considered elsewhere. Finally, it is worthwhile to remark 
briefly that the theory developed here will yields explicit constructions 
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for the Duke-Imamoglu-Ikeda lifting from certain cuspidal automor- 
phic forms on SL 2 (A) to cuspidal automorphic forms on Sp 4i (A), the 
construction via explicit Fourier expansion of this transfer was obtain 
by T. Ikeda in [IkOlj . If combined with the constructions of endoscopy 
correspondences discussed in Section 5, this method is able to con- 
struct the conjectural lifting of Andrianov (|An01J). See also the work 
of Ikeda ( |Ik06] and |Ik08j ) for relevant problems. 

7. Theta Correspondence and (x,6)-Theory 

The theta correspondence for reductive dual pairs is a method, in- 
troduced by R. Howe 1979 ( |Hw79] ) , to construct automorphic forms 
using the theta functions built from the Weil representation on the 
Schrodinger model. The method to use classical theta functions to 
construct automorphic forms goes back at least to the work of C. 
Siegel ( |Sj66l ) and the work of A. W eil ( jWIM] and jWl65] ). It is the 
fundamental work of Howe ( |Hw79j ) which started the representation- 
theoretic approach in this method guided by the idea from Invariant 
Theory. 

The theta correspondence gave the representation-theoretic frame- 
work of the classical theory of modular forms of half-integral weight and 
its relation to the the theory of modular forms of integral weight, i.e. 
the classical Shimura correspondence ( |Sm73] ) . A series of papers of 
J.-L. Waldspurger ( |W80] . |W84] and |W9fj ) gave a complete account 
of this theory, which was beautifully summarized by Piatetski-Shapiro 
in [PS84] . 

One of the early contributions of this method to the theory of au- 
tomorphic forms was the discovery of families of the counter-examples 
of the generalized Ramanujan conjecture for algebraic groups different 
from the general linear groups. The first of such examples was con- 
structed by Howe and Piatetski-Shapiro QHPS79J) and then more ex- 
amples of similar kind were constructed by Piatetski-Shapiro in 1980's 
on GSp(4) including the example of Saito-Kurokawa and by many other 
people for more general groups. The existence of those non-tempered 
cuspidal automorphic representations enriched the spectral theory of 
automorphic forms for general reductive algebraic groups ( |Ar84j and 

ESI]). 

In order to understand the spectral structures of such striking exam- 
ples constructed by the theta correspondence, S. Rallis launched, after 
his series of work joint with G. Schiffmann ( |RS77j . |RS78 ] and |RS81j ) 
and the work of Waldspurger cited above, a systematic investigation 
(sometimes called the Rallis program which was summarized later 
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in ( |R191] )) of the spectral structure of the theta correspondences in 
family (the so-called Rallis Tower Property ( [R184] )). the relation 
between theta correspondence and the Langlands functoriality ( |R184] ) . 
and the characterization of the non-vanishing property of such con- 
structions ( |R187j ). The Rallis program leads to a series of the funda- 
mental contributions in the modern theory of automorphic forms: the 
Rallis Inner Product Formula ( |R187j ). the doubling method for 
the standard L-functions of classical groups by Piatetski-Shapiro and 
Rallis ( |GPSR87p . and the Regularized Siege l-Weil formula of S. 
Kudla and S. Rallis ( |KR88j . |KR94] and [K308] ). 

The regularized Siegel-Weil formula has been extended to all classical 
groups t hroug h the w ork of T. Ikeda ( (Ik96| ). C. Moeglin ( |Mg97|); A . 
Ichino ( [IcOlj . jlcOi] and jlc07]); D . Soudry and the au thor (|J5CT7] ): 
W.-T. Gan and S. Takeda f jGTllj : S, Yaman a ([Yml l] and [Yml2j ): 
and W.-T. Gan, Y.-N. Qiu, and S. Takeda ( |GQT12| . Furthermore, 
the work of W.-T. Gan extends the formula to some exceptional groups 
f fOnOOj . [On08] and [GEE]]). 

Applications of the Rallis program to number theory and arithmetic 
are broad and deep, including arithmeticity of special values of cer- 
tain automorphic L-functions ([HLSk05j, [HLSk06], and |Pr09j for in- 
stance); the nonvanishing of cohomology groups of certain degree over 
Shimura varieties ( |Li92] and |BMMll] ) ; the Kudla program on special 
cycles and a generalized Gross-Zagier formula ( |Kd02j and [YZZ12] ). 
and see also relevant work ( |Hr94] . |Hr07] . |Hr08] . and |HLSnllj ). to 
mention a few. 

7.1. Rallis program. The main idea and method in the Rallis pro- 
gram are explained here only with one case of reductive dual pairs, 
which is completely the same for all other reductive dual pairs. Take 
(Sp 2n , SO m ), which forms a reductive dual pair in Sp 2mn . The theta 
function Q^ a (x) attached to a global Bruhat-Schwartz function <p in 
the Schrodinger module iS(A mn ) of the Weil representation associated 
to the additive character ip a is an automorphic function of moder- 
ate growth on Sp 2mn (A). When restricted to the subgroup Sp 2n (A) x 
SO m (A), the metaplectic double splits. For o G A(SO m ) and n G 
*4(Sp 2n ), if the following integral 

(7.1) / / ef(g,h)^(g)^(h}dgdh 

J Sp 2n (F)\Sp 2n (A) JS0 2m (F)\S0 2m (A) 

converges and is non-zero, then this family of the integrals with all 
G S(A mn ) yields a correspondence between automorphic forms on 
Sp 2 „(A) and automorphic forms on SO m (A). For instance, letting the 
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integer m run, Rallis got the following diagram (called the Witt tower) 
of theta correspondences: 



SO r 



(7.2) / t 

SP2n > S0 4 

\ t 

so 2 

The Rallis tower property of theta correspondence asserts that the 
first occurrence of the theta correspondence of cuspidal automorphic 
representation 7r of Sp 2n (A) in this tower is always cuspidal. Here the 
first occurrence means that for a given automorphic representation 7r 
of Sp 2n (A), the smallest integer m such that the integral (17. ip is non- 
zero for some automorphic representation a of SO m (A) with a certain 
choice of the Bruhat-Schwartz function <p. Hence the spectral theory 
of the theta correspondence is to ask the following two basic questions: 

(1) First Occurrence Problem: For a given irreducible cuspidal 
automorphic representation 7r of Sp 2n (A), how to determine the 
first occurrence of the theta correspondence in the Witt tower 
in terms of the basic invariants attached to 7r? 

(2) Spectral Property of the First Occurrence: For a given 
irreducible cuspidal automorphic representation 7r of Sp 2n (A), 
what kind of cuspidal automorphic representations are expected 
to be constructed at the first occurrence? 

For the First Occurrence Problem, Rallis calculated the L 2 -inner 
product of the image of the theta correspondence, which leads to the 
well-known Rallis Inner Product Formula. This formula expresses 
the L 2 -inner product of the image of the theta correspondence in terms 
of a basic invariant of the original given 7r, that is, the special value 
or residues of the standard automorphic L-function of tt, up to cer- 
tain normalization. The whole program has been completely carried 
out through the work by many people, notably, Rallis ( |R187j and 
fKMj); Kudla and Rallis ( |KR94] ): Moeglin ( |Mg97| ); Lapid and Rallis 
( [LR05] ): Soudry and the author ( [JS07] ); Ginzburg, Soudry and the 
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author QG.IS09] ): Gan and Tak eda ( jGTll] ): Yamana ( |Yml2j ): and 
Gan, Qiu and Takeda ( |GQT12| ). 

The Spectral Property of the First Occurrence is mainly to 
understand the relation of the transfer of automorphic representations 
given by theta correspondence with the Langlands functoriality. 

As suggested by the global Howe duality conjecture ( |Hw79] ) . the 
irreducibility of the first occurrence of the theta correspondences was 
proved by Moeglin (|Mg97|) for even orthogonal group case, by Soudry 
and the author ( |JS07j ) for the odd orthogonal group case, and by 
Chenyan Wu (a current postdoc of the author) in |Wul2j for the unitary 
group case. 

The work of Rallis in 1984 QR184]) started the investigation of the 
relation between the theta correspondence in family and the local Lang- 
lands parameters at unramified local places (see also the work of Kudla 
[Kd86] ) . It seems that the unramified local Langlands parameters can 
not be well preserved through the theta correspondence in family. It 
was J. Adams who first observed that the theta correspondence may 
transfer the Arthur parameters in a way compatible with the Arthur 
conjecture. This is known as the Adams conjecture ([Ad89j), which 
was discussed explicitly for unitary group theta correspondences by 
Harris, Kudla and Sweet in [HKS96]. More recent progress in this as- 
pect is referred to the work of Moeglin ( |Mgll| ). The nature of theta 
correspondence and the Adams conjecture suggest that the theory of 
theta correspondence and related studies in automorphic forms are to 
detect the global Arthur parameter of type (x, b) with a quadratic 
character x of the relevant automorphic representations, and hence it 
seems reasonable to call such studies the (x, &)-theory of automorphic 
representations. 

7.2. (x, &)-theory. Let 7r be an irreducible cuspidal automorphic rep- 
resentation of Sp 2n (A) and let x be a quadratic character of F X \A X . 
The (twisted) standard (partial) L-function L s (s,n x x) was studied 
through the doubling method of Piatetski-Shapiro and Rallis ( [GPSR87] ). 
A combination of the work of Lapid and Rallis ( [LR05] ) and Yamana 
( |Yml2j ) completes the basic theory of this automorphic L-function. 

Following Kudla and Rallis ( KK9 It Theorem 7.2.5]), This partial 
L-function L s (s, ir x x) ma Y have a simple pole at 

So e{l,2,---,[-] + l}. 

If s = sq is a right-most pole of L (s, it x x), then it has the first oc- 
currence a at SO 2n __2s +2; which is an irreducible cuspidal automorphic 
representation of SO2 n -2s +2(A)- According to the local calculation 
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( [Kd86] and [HKS96] . the Adams conjectur e ([Ad89 j and |Mgll| ), and 
the relevant global results ( |GRS97j . [JS07j . |GJS09] and jGJSlll (2)]), 
it is expected that the global Arthur parameter of tt contains (x, b) 
as a simple formal summand for some quadratic character x, which 
should be decided through the choice of the additive character for the 
theta correspondence. In principle, one needs also to consider the Witt 
tower of nonsplit orthogonal groups in order to detect more precisely 
the simple Arthur parameter (x, b) in the global Arthur parameter of 
tt, as suggested by a conjecture of Ginzburg and Gurevich in [GG06J. 

When L s (s, tt x x) is holomorphic for Re(s) > 1, the first occurrence 
of ti at SO m with 2n + 2 < m < An is detected by the nonvanishing 
of L(s,tt x at s = m ~ 2 " , in addition to the local occurrence con- 
ditions following the recent work of Gan, Qiu and Takeda ( |GQT12"1 ). 
If L s (s,tt x x) is holomorphic for Re(s) > 1 for all quadratic char- 
acters X-, then the global Arthur parameter of tt contains no formal 
summand of simple parameters of type (x, b). In this case, the relation 
between (x, b) and the global Arthur parameter of ti should be detected 
through a version of the Gan-Gross-Prasad conjecture f |GGP12"] ). This 
also suggests that one needs to consider a theory to detect the simple 
parameter fa, b) for tt with r G A cusp (a) and a > 1. This is exactly 
the reason to suggest the fa, 6)-theory of automorphic representations, 
which will be discussed below with more details. 



8. Endoscopy Correspondence and fa, 6)-Theory 

As indicated in the (x, 6)-theory discussed above, for a general F- 
quasisplit classical group G n , the relation between the structure of 
global Arthur parameters and the structure of the poles of the partial 
tensor product L-functions L s (s,n x r) can be described as follows, 
where n G A C u SV ,{G n ) and r G ^4 cusp (a). This is the key reason to 
suggest the (r, 6)-theory for automorphic forms of F-quasisplit classical 
groups. 

8.1. Global Arthur parameters and poles of partial tensor prod- 
uct L-functions. Let G be a classical group belonging to £ s \ m (N). 
Each ip G ^2{G) can be written as 

ip := Vi ffl ih ffl • • • A 

where each simple global Arthur parameter = fa, 6&) with Tj G 
*4cus P (cfi) being self-dual and h>l being integers that satisfy certain 
parity condition with Ti, for i = 1, 2, ■ ■ ■ ,r. Now, consider the partial 
tensor product L-functions L (s,tt x r) for some self-dual member in 
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•A cnsp (a). According to the Arthur classification of the discrete spec- 
trum of G, this family of partial L-functions can be defined and has 
meromorphic continuation to the whole complex plane C. 

When ip is a generic parameter, i.e. 6j = 1 for all i, then those partial 
L-functions can be completed through the Arthur-Langlands transfer, 
that is, the ramified local L-functions can also be defined ( |Arl2j ). 
However, when the global parameter ip is not generic, then it is still 
a problem to define the ramified local L-functions for this family of 
global L-functions. On the other hand, the Rankin-Selberg method is 
expected to define the ramified local L-functions for this family of the 
global L-functions. The recent work of Zhang and the author extends 
the work of Ginzburg, Piatetski-Shapiro and Rallis ([GPSR97J) from 
orthogonal groups to the classical groups of hermitian type ( |JZ13j ) and 
the skew-hermit ian case is also considered by Zhang and the author, 
including the work of Xin Shen ([Snl2] and |Snl3j ). extending the work 
of Ginzburg, Rallis, Soudry and the author on the symplectic groups 
and metaplectic groups ( [GJRS11] ). 

In the following discussion, the partial tensor product L-functions are 
used as invariants to make connections to the global Arthur parameters. 
For any tt G n^(e^,), the automorphic L 2 -packet associated to the global 
Arthur parameter ip, define the set: 

(8.1) := {r G U a >i^. cusp (a) | L s (s,n x r) has a pole at s > -}, 

where A cusv (a) denotes the subset of self-dual members in -4 cusp (a). It 
is a non-empty finite set by the Arthur classification theory. Write 

% = {n,r 2 , ■■■ ,r t } 

with n G AuspK) and T; ^ Tj ifi^j- 

For r G r £ n and b > 1 integer, define [r, b] n , which may be called a 
Jordan block of it following Moeglin, to be a pair attached to tc if the 
following properties hold: 

(1) the partial tensor product L-function L s (s,tt x r) is holomor- 
phic for the real part of s greater than and 

(2) L 5 (s, tt x r) has a simple pole at s — 

Note that the right-most pole of the partial tensor product L-function 
L s (s, 7r x r) is at most simple. This follows from the structure of global 
Arthur parameters in ^(G) or the Rankin-Selberg integral method 
(see |JZ12j for instance). 

For each i, take tt^^) to be the Arthur representation of GL a - 6 .(A), 
which are realized as the Speh residues as automorphic representations. 
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It is an easy exercise to check that N = J^* =1 aA if and only if the 
following quotient of partial L-functions 

L S (s, 7T X Ti) 

8.2 -j-^ ^4 

is holomorphic for Re(s) > \ for all i. If there is an % such that the quo- 
tient ( 18. 2 p has a pole at s — 6l,1 2 +1 for some b^\ > 1 and is holomorphic 

for Re(s) > One can define [r^ft^J. By the structure of global 

Arthur parameter ip, 6 i := h and bn have the same parity. Repeating 
this argument, one obtains the following sequence of retypes of tt: 

(8.3) [Ti, bi^] n , [Ti, fe^ljTr, • • ■ , [Ti, fcj,rj7r, 

with the property that b ii0 > b^\ > ■ • ■ > b i)Ti > and the quotient 

L S (S,1T X Tj) 
Y[ V j=0 L S (s, 7r(T<,6i,j) x Ti) 

is holomorphic for Re (s) > ~. Finally the originally given global Arthur 
parameter 0> can be expressed as 

EH*=i ffljLo ijhKj)- 

At this point, the (r, 6)-theory is to detect, for a given 7r e IL^e^), 
the pairs 

(8.4) [n, 6 ii0 ] w , [r 2 , &2,o]7r, • • • , [r t , &t,o]»r, 

by the constructions provided in previous sections. In other words, it is 
to detect the "maximal" simple factors of ip in the sense that a simple 
factor (r, b) of ip is maximal if 

b = max{/3 G Z | (r, /3) is a simple summand of ■0}. 

From the discussion below, it will be much involved in dealing with the 
simple factors of ip, which are not maximal. 

8.2. (r, 6)-towers of endoscopy correspondences. From the con- 
structions discussed in Section 5, if let the integer b vary, one gets the 
following diagrams of constructions of endoscopy correspondences for 
classical groups. 

Consider the case of a = 2e and 2n+l = ab+c with c = 2/+1. When 
r is of symplectic type, take b = 21 + 1 to be odd. Conjecture 5.4 can 
be more specified at this case by the following diagram of endoscopy 
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corresp ondences : 



^so c ffl(r,3) 



ipso c ffl (r, 1) 



4>so c B (r, 1) 



Here the parameter ?/>so c B ( r , 1) is understood in the sense of the 
Grothendieck group generated by the global Arthur parameters, which 
also gives the meaning for general ip\ B ip%. When r is of orthogonal 
type, take b = 21 to be even. Conjecture 5.4 can be more specified by 
the following diagram of endoscopy correspondences: 



^so c B (r, 2) 



^so c B (r, 1) 



^so c B(r,2) 



The constructions discussed in Section 5.3 will produce similar di- 
agrams of endoscopy correspondences for other above. The 
details will be omitted here. 

It is important to note that there are also two types of basic tri- 
angles of constructions here in Diagrams (18.51) and (18. 6p . which is a 
generalization of the two types of the basic triangles of constructions 



/ t 

(8.5) Vso c SO c — ► SO c+ , 

\ t 

so c _ 



SO c+ 2a 



.6) 



/ t 
^so c SO c — > SO c 

\ t 

SO c _2a 
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discussed in Section 6. The properties of these basic triangles of con- 
structions are the key to establish the functorial transfer properties of 
the general diagrams of constructions in ( 18. 5p and ( 18. 6p . 

Next, consider the constructions discussed in Section 5.4. 

Take a = 2e + 1 with c = 2f + 1 and 2n = ab + c—1. In this case, r 
must be of orthogonal type and 6 = 2/ must be even. A combination of 
the constructions discussed in Conjectures 5.10 and 5.12 can be more 
specified by the following diagram of endoscopy correspondences: 



Sp 2 /+2a ^so 2/+1 ffl (r, 2) 

/* t 

• 7 ) ^so 2/+1 SO2/+1 — > Sp 2/ ^so 2/+1 fflO,0) 

\ t 

Sp 2 /-2a ^so 2/+1 B (r, 2) 



Of course, a different combination of the constructions discussed in 
Conjectures 5.10 and 5.12 can be more specified by the following dia- 
gram of endoscopy correspondences: 



S0 2/+1+2a ^ 2/ ffl(r,2) 
/ t 

^sp 2f Sp 2/ — ► S0 2/+ i ^ 2/ ffl(r,0) 

\ T 

S0 2/+1 _ 2a % 2/ B(r,2) 
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Note that if take a = 1 and (r, b) = (x, b), then the diagram (18.7p gives 
one of the Witt towers for the theta correspondences. Since the auto- 
morphic kernel functions used for the endoscopy correspondences 
are defined in a different way, it is necessary to check that both the 
endoscopy correspondences constructed here and the theta correspon- 
dences do produce the same correspondences when the towers coincide. 
This is in fact a way to verify the conjecture of Adams on the compati- 
bility of theta correspondences and the Arthur parameters as discussed 
in the (x, 6)-theory in Section 7. 

Similarly, various combinations of the constructions in Conjectures 
5.9 and 5.11 produce diagrams of endoscopy correspondences which 
may also specialized to the theta correspondences for reductive dual 
pairs (SO2/, Sp 2n ) and pairs of unitary groups. 

8.3. (r, 6)-theory of automorphic forms. The (r, 6)-theory of au- 
tomorphic forms is to establish the spectral property of the diagrams 
of endoscopy correspondences as discussed in the previous section and 
verify the conjectures stated in Sections 5.3 and 5.4; and detect the 
structure of global Arthur parameters for a given irreducible cuspi- 
dal automorphic representation tt of an F-quasisplit classical group in 
terms of the basic invariants attached to 71. In short, this is to estab- 
lish a version of the Rallis program in this generality and look for the 
analogy of the Rallis inner product formula for the endoscopy corre- 
spondences and for a replacement of the Siegel-Weil formula for this 
general setting. An important application of those constructions has 
been found in a recent work of Lapid and Mao ( |LM12j . which can be 
viewed as part of the (r, 6)-theory. 

9. Final Remarks 

There are a few closely related topics, which will be briefly discussed 
here and the more details of which may be considered elsewhere. 

9.1. On generic global Arthur parameters. A global Arthur pa- 
rameter if) G ^2{G) is called generic if it has form 

V = (r 1 ,l)ffl(r 2 ,l)ffl...ffl(r n l) 

with n G -4. CU sp( a i) being conjugate self-dual. The special cases where 
b = 1 and ip2 is generic of conjectures in Sections 5.3 and 5.4 yield the 
constructions of endoscopy correspondences for the decomposition of 
the generic global Arthur parameter ijj = (r, 1) EH ip2- For G G £ S i m (N), 
if one only concerns the endoscopy transfers for irreducible generic cus- 
pidal automorphic representations of G(A), there is an indirect way 
to prove the existence of such a transfer, by combining the general 
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constructions of automorphic descents ( |GRSllj ) with the Langlands 
functorial transfer from the classical group G to the general linear group 
GL(iV) for irreducible generic cuspidal automorphic representations of 
G(A) via the method of Co nverse T heorem QCKPSS04] and jCPSSllj ), 
as explained in the book ( |GRSllj ). In 2008, Ginzburg in |G08] gave 
a framework to constructing endoscopy transfers for generic cuspidal 
automorphic representations for F-split classical groups, and sketched 
proofs for some cases, which may not be applied to all the cases because 
of technical reasons. The constructions described in this paper extends 
the framework of |G08j to cover all possible cases. The on-going work 
of Lei Zhang and the author is dealing with the case of F-quasisplit 
classical groups of hermitian type ( |JZ13j ). 

For irreducible generic cuspidal automorphic representations rr of 
G(A), it is a basic question to know the structure of their global Arthur 
parameters in terms of the basic invariants attached to ir. It is clear 
that one needs invariants in addition to the Fourier coefficients. One 
choice is to use the set % n as introduced in (8.1), which is to use the 
poles of the tensor product L-functions L s (s, n x r) with r G *4 C usp( a )- 
Note that the poles occur at s = 1 since it is generic. 

From the point of view of the Langlands Beyond Endoscopy ( |L04j ) . 
it is better to have a characterization of the endoscopy structure in 
terms of the order of the pole at s = 1 of a family of automorphic L- 
functions L s (s,7r,p) for some complex representations p of the Lang- 
lands dual group L G. Such a characterization is fully discussed by 
the author in |Jn06j for irreducible generic cuspidal automorphic rep- 
resentation of G = S0 2n +i with p going through all the fundamental 
representations of the dual group Sp 2n (C). The results can be naturally 
extended to all classical groups. 

Another characterization of the endoscopy structure in terms of peri- 
ods of automorphic forms is formulated as conjectures in a recent work 
of Ginzburg and the author in [GJ12J for G = S02 n +i, where some 
lower rank cases have been verified. The formulation of the conjectures 
works for all classical groups. The details are omitted here. 

9.2. On the local theory. Just as in the theory of theta correspon- 
dences, it is interesting to consider the local theory of theta correspon- 
dences and the local-global relations of the theta correspondences for 
automorphic forms. For the (r, 6)-theory, one may consider the local 
(r, &)-theory with r an irreducible, essentially tempered representation 
of GL a over a local field. This local theory should include the con- 
structions of local endoscopy correspondences and other related topics 
as the extension of the current local theory of theta correspondences 
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and related topics. In this aspect, one may consider the theory of local 
descent as a preliminary step of the local (r, 6)-theory, which has been 
discussed in the work of Soudry and the author ( [JS03] . | JS04| . and 
[JST2]) in the work of C.-F. Nien, Y.-J. Qin and the au thor (|JN^08l , 
|JNQ10| , and |JNQ12|), and the work of Baiying Liu ( |Liull| ) and of 
C. Jantzen and Liu ( |JaL12j ) over the p-adic local fields. Over the 
archimedean local fields, the recent work of A. Aizenbud, D. Goure- 
vitch, and S. Sahi ( |AGSllj . of Gourevitch and Sahi in |GS12j . and of 
B. Harris in |H12j are relevant to the theme of this paper. However, 
the local (r, 6)-theory for archimedean local fields is open. 



9.3. Constructions of different types. When the Weil representa- 
tion is replaced by the minimal representation of exceptional groups, 
the local theory of theta correspondences has been extended to the 
so-called exceptional theta correspondences, which works both locally 
and globally. The local theory started from the work of D. Kazhdan 
and G. Savin in }KzS90] and the global theory is referred to the work 
of Ginzburg, Rallis and Soudry in [GRS97J. Among many contributors 
to the theory locally and globally are Kazhdan, Savin, Rubenthaler, 
Ginzburg, Rallis, Soudry, J.-S. Li, Gross, Gan, Gurevich, the author 
and others. It should be mentioned that important applications of the 
exceptional theta correspondences include the work of Ginzburg, Rallis 
and Soudry on the theory of cubic Shimura correspondences ( |GRS97j ). 
the work of Gan, Gurevich and the author on cuspidal spectrum of G2 
with high multiplicity (|GGJ02J), and the work of Gross and Savin on 
the existence of motivic Galois group of type G2 QGrS98]), to mention 
a few. The (r, 6)-theory for exceptional groups is essential not known. 

Finally, it is to mention the work of Ginzburg, Soudry and the author 
on the partial automorphic descent constructions for GL 2 in |GJS11] . 
which is expected to be in a different nature comparing to the whole 
theory discussed here. A more general consideration of the construc- 
tions of this nature will be discussed in a future work. 
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